Oct 31 02:54:37.468334 ip-10-0-129-107 systemd[1]: Starting Kubernetes Kubelet... Oct 31 02:54:37.871818 ip-10-0-129-107 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:54:37.871818 ip-10-0-129-107 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 02:54:37.871818 ip-10-0-129-107 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:54:37.871818 ip-10-0-129-107 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:54:37.876398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876320 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:37.876398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876397 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876402 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876405 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876408 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876411 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876413 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876416 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876419 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876421 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876424 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876426 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876429 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876431 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876435 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876438 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876440 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876443 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876446 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876448 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876451 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:37.876465 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876453 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876456 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876458 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876461 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876464 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876469 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876473 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876476 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876478 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876481 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876484 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876486 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876489 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876492 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876494 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876497 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876500 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876502 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876505 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:37.876926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876508 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876510 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876513 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876515 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876518 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876520 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876523 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876525 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876527 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876530 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876532 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876535 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876538 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876541 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876543 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876546 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876549 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876552 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876555 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:37.877414 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876557 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876560 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876563 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876565 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876568 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876570 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876573 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876575 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876578 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876580 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876582 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876585 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876587 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876590 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876592 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876595 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876597 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876600 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876603 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:37.877858 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876605 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876608 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876610 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876613 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876615 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876618 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876621 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876625 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876627 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876631 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876634 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876636 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876639 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876642 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876645 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876647 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876651 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876656 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876659 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876663 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:37.878358 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876665 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876668 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876671 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876673 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876676 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876680 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876684 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.876687 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877433 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877439 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877465 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877470 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877474 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877477 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877480 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877483 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877485 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877489 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877491 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:37.878903 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877494 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877496 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877499 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877501 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877504 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877507 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877509 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877512 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877515 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877518 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877520 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877523 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877525 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877528 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877531 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877534 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877537 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877539 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877543 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877546 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:37.879372 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877549 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877551 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877554 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877557 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877560 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877562 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877565 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877568 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877571 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877573 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877576 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877578 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877581 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877584 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877586 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877589 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877591 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877594 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877596 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:37.879848 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877599 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877602 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877605 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877608 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877612 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877616 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877619 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877622 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877625 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877627 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877630 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877632 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877635 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877637 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877640 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877642 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877648 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877650 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877653 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:37.880303 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877656 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877659 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877661 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877664 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877666 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877669 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877671 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877674 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877676 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877679 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877682 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877684 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877687 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877689 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877692 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877696 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877698 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877701 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877703 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877706 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:37.880775 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877708 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877711 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877714 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877717 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877719 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877722 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877725 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877728 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877731 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877735 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877737 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877741 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877744 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877746 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877748 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877751 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.877753 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878284 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878289 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:37.881263 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878292 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878295 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878298 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878300 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878303 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878306 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878308 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878311 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878314 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878317 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878319 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878322 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878325 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878327 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878330 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878332 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878337 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878339 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878342 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878344 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:37.881714 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878347 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878349 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878353 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878356 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878359 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878362 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878364 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878367 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878370 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878373 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878376 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878378 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878392 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878396 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878400 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878402 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878405 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878408 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878411 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:37.882213 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878413 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878416 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878419 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878421 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878424 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878426 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878429 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878432 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878434 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878437 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878440 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878443 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878445 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878448 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878451 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878454 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878457 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878460 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878462 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878464 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:37.882655 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878467 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878469 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878473 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878476 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878480 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878482 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878485 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878488 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878490 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878492 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878495 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878497 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878500 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878504 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878507 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878510 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878513 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878515 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878519 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:37.883127 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878522 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878524 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878528 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878531 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878533 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878536 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878539 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878542 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878546 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878548 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878550 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878553 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878555 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878558 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878560 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878563 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878565 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878568 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878570 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:37.883566 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878574 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878576 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878579 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878582 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878585 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878587 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.878590 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879673 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879683 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879690 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879696 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879700 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879703 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879708 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879712 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879715 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879718 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879722 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879725 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879728 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879732 2577 flags.go:64] FLAG: --cgroup-root="" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879736 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879739 2577 flags.go:64] FLAG: --client-ca-file="" Oct 31 02:54:37.884008 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879742 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879745 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879749 2577 flags.go:64] FLAG: --cluster-domain="" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879752 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879755 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879758 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879762 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879766 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879769 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879772 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879776 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879778 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879781 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879784 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879788 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879790 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879794 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879798 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879801 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879804 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879807 2577 flags.go:64] FLAG: --enable-server="true" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879810 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879815 2577 flags.go:64] FLAG: --event-burst="100" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879818 2577 flags.go:64] FLAG: --event-qps="50" Oct 31 02:54:37.884566 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879821 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879824 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879827 2577 flags.go:64] FLAG: --eviction-hard="" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879831 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879834 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879837 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879841 2577 flags.go:64] FLAG: --eviction-soft="" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879844 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879847 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879850 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879853 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879856 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879859 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879862 2577 flags.go:64] FLAG: --feature-gates="" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879866 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879869 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879872 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879876 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879878 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879882 2577 flags.go:64] FLAG: --help="false" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879887 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879890 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879893 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 02:54:37.885168 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879896 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879899 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879903 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879906 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879909 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879911 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879915 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879918 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879922 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879924 2577 flags.go:64] FLAG: --kube-reserved="" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879927 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879930 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879933 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879936 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879939 2577 flags.go:64] FLAG: --lock-file="" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879942 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879945 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879948 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879954 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879956 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879959 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879962 2577 flags.go:64] FLAG: --logging-format="text" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879965 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879969 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 02:54:37.885730 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879971 2577 flags.go:64] FLAG: --manifest-url="" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879974 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879979 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879982 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879986 2577 flags.go:64] FLAG: --max-pods="110" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879990 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879993 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879996 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.879999 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880002 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880005 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880008 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880015 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880018 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880022 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880026 2577 flags.go:64] FLAG: --pod-cidr="" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880030 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880042 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880045 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880048 2577 flags.go:64] FLAG: --port="10250" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880052 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880055 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-04b499806d0e3d430" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880058 2577 flags.go:64] FLAG: --qos-reserved="" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880061 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880064 2577 flags.go:64] FLAG: --register-node="true" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880067 2577 flags.go:64] FLAG: --register-with-taints="" Oct 31 02:54:37.886343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880071 2577 flags.go:64] FLAG: --registry-burst="10" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880074 2577 flags.go:64] FLAG: --registry-qps="5" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880077 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880080 2577 flags.go:64] FLAG: --reserved-memory="" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880083 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880087 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880090 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880093 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880096 2577 flags.go:64] FLAG: --runonce="false" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880099 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880102 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880106 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880109 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880112 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880115 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880118 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880122 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880124 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880127 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880130 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880133 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880136 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880140 2577 flags.go:64] FLAG: --system-cgroups="" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880143 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880148 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 02:54:37.886981 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880151 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880154 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880158 2577 flags.go:64] FLAG: --tls-min-version="" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880161 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880163 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880166 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880169 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880172 2577 flags.go:64] FLAG: --v="2" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880177 2577 flags.go:64] FLAG: --version="false" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880181 2577 flags.go:64] FLAG: --vmodule="" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880185 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880188 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880327 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880332 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880335 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880337 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880340 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880343 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880347 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880350 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880352 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880356 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:37.887607 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880360 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880363 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880365 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880368 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880370 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880373 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880376 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880378 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880381 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880385 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880389 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880393 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880396 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880399 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880402 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880405 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880407 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880410 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880413 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:37.888147 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880415 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880418 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880420 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880423 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880425 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880428 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880430 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880433 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880436 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880440 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880442 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880445 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880448 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880450 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880453 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880455 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880459 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880462 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880465 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:37.888603 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880467 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880470 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880472 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880475 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880478 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880480 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880483 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880486 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880488 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880491 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880493 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880496 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880498 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880501 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880503 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880505 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880508 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880510 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880513 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880516 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:37.889060 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880518 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880521 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880525 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880527 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880530 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880533 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880536 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880538 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880541 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880545 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880547 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880549 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880552 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880555 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880557 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880560 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880565 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880568 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880571 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880573 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:37.889576 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880576 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880579 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880581 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880583 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880586 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880588 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880591 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880593 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880596 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880598 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880601 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880603 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880605 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880608 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880612 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880614 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880617 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:37.890115 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880620 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.880646 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880680 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880684 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880689 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880692 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880695 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880698 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880700 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880703 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880706 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880708 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880711 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880714 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880717 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880719 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:37.890521 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880721 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880724 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880727 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880729 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880732 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880734 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880737 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880739 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880742 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880745 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880747 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880750 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880752 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880755 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880759 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880762 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880764 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880767 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880769 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880772 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:37.890893 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880776 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880778 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880781 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880784 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880787 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880789 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880792 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880795 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880797 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880800 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880803 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880806 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880808 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880811 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880813 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880816 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880818 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880821 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880823 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:37.891509 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880826 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880828 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880831 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880833 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880836 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880839 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880841 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880845 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880848 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880850 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880853 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880855 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880858 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880863 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880867 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880870 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880873 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880875 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880878 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:37.892290 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880880 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880883 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880886 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880889 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880893 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880897 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880901 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880904 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880906 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880909 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880911 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880913 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880916 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880919 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880922 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880924 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880927 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880929 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880932 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:37.892814 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880934 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880939 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880942 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880945 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880948 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880950 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880953 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880957 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880959 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880962 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880964 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880967 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880970 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880972 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.880974 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.887866 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.887881 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.888606 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.888612 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888670 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:37.893262 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888675 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888678 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888681 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888684 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888687 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888690 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888692 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888695 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888698 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888701 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888704 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888706 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888709 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888712 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888715 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888717 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888720 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888723 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888725 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:37.893727 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888728 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888731 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888733 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888736 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888738 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888741 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888743 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888746 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888748 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888751 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888753 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888756 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888759 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888762 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888765 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888767 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888770 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888773 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888775 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:37.894222 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888778 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888781 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888783 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888786 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888789 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888791 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888794 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888797 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888799 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888802 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888805 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888807 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888810 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888813 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888815 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888817 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888820 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888824 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888827 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888830 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:37.894659 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888832 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888835 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888837 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888840 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888843 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888846 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888848 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888851 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888854 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888856 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888859 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888861 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888864 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888867 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888869 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888872 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888874 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888877 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888880 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:37.895203 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888882 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888886 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888890 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888892 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888895 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888897 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888900 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888902 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888905 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888907 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888910 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888913 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888915 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888918 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888920 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888923 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888925 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888928 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888931 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:37.895669 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888933 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888936 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888938 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888941 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888943 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888946 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888948 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888951 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.888953 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.888982 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889136 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889140 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889144 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889147 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889150 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889153 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:37.896139 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889155 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889158 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889160 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889163 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889165 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889168 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889170 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889173 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889175 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889178 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889180 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889183 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889185 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889188 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889191 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889193 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889196 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889198 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889201 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:37.896515 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889204 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889206 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889209 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889211 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889214 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889216 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889219 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889221 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889224 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889226 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889229 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889233 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889237 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889240 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889245 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889247 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889250 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889253 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889256 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:37.896963 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889258 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889261 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889263 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889266 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889269 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889271 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889274 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889276 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889279 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889281 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889284 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889286 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889289 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889291 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889294 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889297 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889299 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889302 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889304 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889307 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:37.897476 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889310 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889312 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889315 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889317 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889321 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889324 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889326 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889330 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889333 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889336 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889338 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889341 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889344 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889346 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889349 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889351 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889354 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889356 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889359 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889361 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:37.897932 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889364 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889366 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889369 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889371 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889374 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889376 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889379 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889381 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889384 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889386 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889389 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889392 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889396 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889399 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889402 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889405 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889407 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889410 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889413 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:37.898480 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889417 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889420 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889422 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.889452 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889492 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889496 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889499 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889502 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889505 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889508 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889511 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889513 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889516 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889519 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889521 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:37.898923 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889524 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889526 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889529 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889532 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889534 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889537 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889539 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889542 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889547 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889551 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889554 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889557 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889559 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889562 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889565 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889567 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889570 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889573 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889577 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:37.899439 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889580 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889583 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889586 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889588 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889591 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889594 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889596 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889599 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889601 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889604 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889606 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889610 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889612 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889615 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889618 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889620 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889623 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889625 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889628 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889630 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:37.899949 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889633 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889639 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889641 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889644 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889647 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889649 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889652 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889655 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889657 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889660 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889663 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889667 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889669 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889672 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889674 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889677 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889680 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889682 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889685 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889687 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:37.900454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889690 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889692 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889695 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889697 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889700 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889702 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889705 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889708 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889712 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889715 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889718 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889721 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889724 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889728 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889730 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889733 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889735 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889738 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889740 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:37.900926 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889743 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889746 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889748 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889751 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889758 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889761 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889763 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889766 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889769 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889771 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889774 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889776 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889779 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889782 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889785 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889787 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:37.889790 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.889840 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.893889 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 02:54:37.901398 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.894973 2577 server.go:1014] "Starting client certificate rotation" Oct 31 02:54:37.901862 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.895072 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 02:54:37.901862 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.895946 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 02:54:37.901862 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.897266 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:54:37.901862 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.898356 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:54:37.913351 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.913333 2577 log.go:25] "Validated CRI v1 runtime API" Oct 31 02:54:37.916978 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.916966 2577 log.go:25] "Validated CRI v1 image API" Oct 31 02:54:37.917773 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.917741 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 02:54:37.920816 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.920792 2577 fs.go:135] Filesystem UUIDs: map[049aa447-5ff1-4a9a-a3a6-cb611b7c2f58:/dev/nvme0n1p4 0f0aad77-171b-4b4a-a9b8-d2c86d4ebc2d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 02:54:37.920907 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.920813 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 02:54:37.927369 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.927240 2577 manager.go:217] Machine: {Timestamp:2026-10-31 02:54:37.924634971 +0000 UTC m=+0.361296987 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100465 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec238c92b3df410465943d5f9dfa6d2d SystemUUID:ec238c92-b3df-4104-6594-3d5f9dfa6d2d BootID:933e9d3d-8dae-4436-9596-8a6f7be3b816 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:82:5a:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:82:5a:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:c8:06:12:c0:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 02:54:37.927369 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.927362 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 02:54:37.927545 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.927527 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 02:54:37.931210 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.931188 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 02:54:37.931351 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.931212 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-107.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 02:54:37.931425 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.931353 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 02:54:37.931425 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.931362 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 02:54:37.931425 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.931410 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:54:37.931501 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.931428 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:54:37.932645 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.932631 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 02:54:37.936407 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.936392 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 02:54:37.937177 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.937165 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 02:54:37.937296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.937287 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 02:54:37.937330 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.937319 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 31 02:54:37.937357 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.937330 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 02:54:37.937357 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.937340 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 02:54:37.937357 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.937349 2577 kubelet.go:405] "Adding apiserver pod source" Oct 31 02:54:37.937357 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.937357 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 02:54:37.937728 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.937713 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:54:37.941877 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.941853 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 02:54:37.947942 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.947927 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:54:37.948763 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.948748 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 02:54:37.948828 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.948788 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 02:54:37.948988 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.948975 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 02:54:37.949063 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.948996 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 02:54:37.949063 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.949005 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 02:54:37.949063 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.949015 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 02:54:37.949063 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.949024 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 02:54:37.949063 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.949048 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 02:54:37.949789 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.949779 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 02:54:37.949839 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.949794 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 02:54:37.949839 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.949806 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 02:54:37.949839 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.949815 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 02:54:37.949839 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.949828 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 02:54:37.950061 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.949841 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 02:54:37.951565 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.951553 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 02:54:37.951630 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.951573 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 02:54:37.955430 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.955414 2577 server.go:1263] "Started kubelet" Oct 31 02:54:37.955545 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.955490 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 02:54:37.955545 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.955514 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 02:54:37.955629 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.955575 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 02:54:37.956153 ip-10-0-129-107 systemd[1]: Started Kubernetes Kubelet. Oct 31 02:54:37.957135 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.956933 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 02:54:37.957517 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.957486 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 31 02:54:37.960269 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.959073 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-107.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 02:54:37.964231 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:37.964207 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-107.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 02:54:37.964318 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:37.964237 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 02:54:37.964742 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:37.964724 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 02:54:37.965173 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:37.964081 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-107.us-east-2.compute.internal.18e37db1814a179e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-107.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-107.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 02:54:37.955389342 +0000 UTC m=+0.392051356,LastTimestamp:2026-10-31 02:54:37.955389342 +0000 UTC m=+0.392051356,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-107.us-east-2.compute.internal,}" Oct 31 02:54:37.966656 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.966637 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 02:54:37.967265 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.967252 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 02:54:37.967450 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:37.967429 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:37.967971 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.967953 2577 factory.go:55] Registering systemd factory Oct 31 02:54:37.968069 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.967972 2577 factory.go:223] Registration of the systemd container factory successfully Oct 31 02:54:37.968069 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.967994 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 02:54:37.968069 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.967996 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 02:54:37.968069 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.968017 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 02:54:37.968247 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.968095 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 31 02:54:37.968247 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.968105 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 31 02:54:37.968247 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.968230 2577 factory.go:153] Registering CRI-O factory Oct 31 02:54:37.968247 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.968245 2577 factory.go:223] Registration of the crio container factory successfully Oct 31 02:54:37.968412 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.968291 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 02:54:37.968412 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.968309 2577 factory.go:103] Registering Raw factory Oct 31 02:54:37.968412 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.968323 2577 manager.go:1199] Started watching for new ooms in manager Oct 31 02:54:37.969520 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.969441 2577 manager.go:319] Starting recovery of all containers Oct 31 02:54:37.970651 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.970381 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-75hth" Oct 31 02:54:37.973184 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:37.973150 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-107.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 02:54:37.976857 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.976833 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-75hth" Oct 31 02:54:37.982764 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.982746 2577 manager.go:324] Recovery completed Oct 31 02:54:37.985751 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:37.985721 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 02:54:37.986959 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.986947 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:37.988657 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.988640 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:37.988727 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.988675 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:37.988727 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.988691 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:37.989302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.989286 2577 cpu_manager.go:226] "Starting" policy="none" Oct 31 02:54:37.989302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.989299 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:37.989451 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.989299 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 02:54:37.989451 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.989341 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 02:54:37.990093 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.990077 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:37.990164 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.990108 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:37.990164 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.990124 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:37.991447 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.991431 2577 policy_none.go:50] "Start" Oct 31 02:54:37.991518 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.991450 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 02:54:37.991518 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.991463 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 02:54:37.993567 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:37.993553 2577 policy_none.go:44] "Start" Oct 31 02:54:38.043740 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.043703 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 02:54:38.046839 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.046824 2577 manager.go:341] "Starting Device Plugin manager" Oct 31 02:54:38.046920 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.046870 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 02:54:38.046920 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.046881 2577 server.go:87] "Starting device plugin registration server" Oct 31 02:54:38.047167 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.047154 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 02:54:38.047204 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.047169 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 02:54:38.047301 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.047275 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 02:54:38.047396 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.047362 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 02:54:38.047396 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.047371 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 02:54:38.047833 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.047813 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 02:54:38.047901 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.047867 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.096482 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.096455 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 02:54:38.098457 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.097658 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 02:54:38.098457 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.097677 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 02:54:38.098457 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.097716 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 02:54:38.098457 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.097750 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 02:54:38.105833 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.105811 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:38.147986 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.147924 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:38.148948 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.148928 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:38.149020 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.148963 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:38.149020 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.148973 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:38.149020 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.149012 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.157746 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.157731 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.157809 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.157753 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-107.us-east-2.compute.internal\": node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.171525 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.171507 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.198790 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.198771 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal"] Oct 31 02:54:38.198853 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.198819 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:38.199687 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.199674 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:38.199739 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.199702 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:38.199739 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.199711 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:38.201916 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.201903 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:38.202087 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.202073 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.202143 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.202100 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:38.202659 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.202637 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:38.202659 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.202653 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:38.202779 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.202668 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:38.202779 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.202675 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:38.202779 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.202683 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:38.202779 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.202685 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:38.204624 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.204610 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.204699 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.204633 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:38.205344 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.205325 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:38.205436 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.205355 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:38.205436 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.205367 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:38.224287 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.224267 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" node="ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.227186 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.227168 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" node="ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.269692 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.269671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.269768 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.269698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.269768 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.269721 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9a871c29174a296dbd1f4a73e3997d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"da9a871c29174a296dbd1f4a73e3997d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.271640 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.271626 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.370684 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.370661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.370684 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.370685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9a871c29174a296dbd1f4a73e3997d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"da9a871c29174a296dbd1f4a73e3997d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.370844 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.370700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.370844 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.370755 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.370844 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.370774 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.370844 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.370799 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9a871c29174a296dbd1f4a73e3997d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"da9a871c29174a296dbd1f4a73e3997d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.371713 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.371698 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.472159 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.472092 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.528281 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.528256 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.530412 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.530396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:38.573122 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.572970 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.673377 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.673353 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.773872 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.773806 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.874269 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.874240 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.894399 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.894377 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 02:54:38.894513 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.894496 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 02:54:38.974400 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:38.974380 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:54:38.984314 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.984285 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 02:49:38 +0000 UTC" deadline="2028-04-20 12:08:22.317497332 +0000 UTC" Oct 31 02:54:38.984314 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:38.984313 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12897h13m43.333188445s" Oct 31 02:54:39.051231 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.051198 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:39.068404 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.068371 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:39.081005 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.080983 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:54:39.082780 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.082766 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:54:39.092279 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.092261 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:54:39.153972 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:39.153943 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f3d03d6994989fb7ae18d976e5b40a.slice/crio-fcd97591b2f336f4047f85efd77085896f09401c9144412995da9b3073c7a659 WatchSource:0}: Error finding container fcd97591b2f336f4047f85efd77085896f09401c9144412995da9b3073c7a659: Status 404 returned error can't find the container with id fcd97591b2f336f4047f85efd77085896f09401c9144412995da9b3073c7a659 Oct 31 02:54:39.161103 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.161085 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 02:54:39.164862 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:39.164841 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9a871c29174a296dbd1f4a73e3997d.slice/crio-5e4788a9e9512eb0a3edd7845396e8b3882e48d761abb29316c78bd9543d9527 WatchSource:0}: Error finding container 5e4788a9e9512eb0a3edd7845396e8b3882e48d761abb29316c78bd9543d9527: Status 404 returned error can't find the container with id 5e4788a9e9512eb0a3edd7845396e8b3882e48d761abb29316c78bd9543d9527 Oct 31 02:54:39.331426 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.331344 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:39.439225 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.439191 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:39.938724 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.938691 2577 apiserver.go:52] "Watching apiserver" Oct 31 02:54:39.945691 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.945659 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 02:54:39.946052 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.946008 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-ctx6n","kube-system/konnectivity-agent-khwpz","kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal","openshift-image-registry/node-ca-xlsrl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-n8slw","openshift-ovn-kubernetes/ovnkube-node-vhcwk","kube-system/global-pull-secret-syncer-vtlr6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn","openshift-dns/node-resolver-mxcvd","openshift-multus/multus-additional-cni-plugins-g9pb4","openshift-multus/multus-qhcg9","openshift-network-operator/iptables-alerter-tq4b7","openshift-network-diagnostics/network-check-target-kjmq9"] Oct 31 02:54:39.949000 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.948968 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.951796 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.951775 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:39.952491 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.952472 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 02:54:39.952881 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.952862 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-49z5f\"" Oct 31 02:54:39.953031 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.953010 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 02:54:39.953137 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.953121 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 02:54:39.953186 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.953126 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 02:54:39.953186 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.953169 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 02:54:39.953827 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.953808 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 02:54:39.953925 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.953876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:39.953994 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:39.953938 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:54:39.954813 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.954701 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 02:54:39.954813 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.954779 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 02:54:39.955331 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.955247 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2b67f\"" Oct 31 02:54:39.955331 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.955291 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 02:54:39.955496 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.955349 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 02:54:39.955496 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.955347 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 02:54:39.956401 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.956380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:39.959204 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.958966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:39.959204 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:39.959037 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:54:39.962564 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.962286 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:39.962564 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:39.962358 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:54:39.965043 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.965025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:39.967539 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.967251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:39.967670 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.967652 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 02:54:39.969718 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.969698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:54:39.975086 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.975060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:39.977628 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.977613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:39.978301 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-netns\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.978465 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:39.978590 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:39.978590 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-device-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:39.978590 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978524 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2be3a160-6ada-4e51-a6bf-19745a8670f4-host\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:39.978590 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxm7m\" (UniqueName: \"kubernetes.io/projected/2be3a160-6ada-4e51-a6bf-19745a8670f4-kube-api-access-jxm7m\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:39.978590 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-ovn\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.978859 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-node-log\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.978859 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978629 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:39.978859 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-agent-certs\") pod \"konnectivity-agent-khwpz\" (UID: \"24d5a9bc-29a3-42b5-bf90-9468b578b0f7\") " pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:54:39.978859 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-log-socket\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.978859 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978760 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:39.978859 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sskj6\" (UniqueName: \"kubernetes.io/projected/af399405-2475-49b9-9fdd-7f43dff044b6-kube-api-access-sskj6\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:39.978859 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-kubelet\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-bin\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.978956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-netd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovn-node-metrics-cert\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-script-lib\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-system-cni-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-sys-fs\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-systemd-units\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979207 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-systemd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979266 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-env-overrides\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:39.979349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fd7955b-02fe-411e-83a4-abac6fdbb242-iptables-alerter-script\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fd7955b-02fe-411e-83a4-abac6fdbb242-host-slash\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-kubelet-config\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4vt\" (UniqueName: \"kubernetes.io/projected/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-kube-api-access-lx4vt\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-cnibin\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-249fn\" (UniqueName: \"kubernetes.io/projected/4249fb31-38de-4684-abd0-6d49c6dacd59-kube-api-access-249fn\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9pnv\" (UniqueName: \"kubernetes.io/projected/24901959-cd66-4412-a070-aeb1cfaa8444-kube-api-access-g9pnv\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-socket-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdx4z\" (UniqueName: \"kubernetes.io/projected/6fd7955b-02fe-411e-83a4-abac6fdbb242-kube-api-access-zdx4z\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-dbus\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-var-lib-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2be3a160-6ada-4e51-a6bf-19745a8670f4-serviceca\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979784 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-konnectivity-ca\") pod \"konnectivity-agent-khwpz\" (UID: \"24d5a9bc-29a3-42b5-bf90-9468b578b0f7\") " pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:54:39.980293 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-slash\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.981131 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-etc-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.981131 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-config\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:39.981131 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.979923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-registration-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:39.981131 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.980166 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qhcg9" Oct 31 02:54:39.983341 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:39.983316 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 02:54:40.007680 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.007644 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-584x7" Oct 31 02:54:40.018869 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.018843 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-584x7" Oct 31 02:54:40.065053 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.065009 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 02:54:40.065409 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.065391 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 02:54:40.065819 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.065801 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 02:54:40.065890 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.065847 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:54:40.065947 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.065904 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 02:54:40.066003 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.065983 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wr2d\"" Oct 31 02:54:40.066127 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.066102 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 02:54:40.066256 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.066197 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wcv7f\"" Oct 31 02:54:40.066327 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.066265 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 02:54:40.066436 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.066411 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x5rg4\"" Oct 31 02:54:40.066498 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.066478 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 02:54:40.066557 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.066540 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 02:54:40.066831 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.066813 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 02:54:40.066997 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.066899 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 02:54:40.067069 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.067021 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 02:54:40.067199 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.067137 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:54:40.067269 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.067201 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sts47\"" Oct 31 02:54:40.067269 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.067241 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 02:54:40.067373 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.067335 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tbpkr\"" Oct 31 02:54:40.067373 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.067371 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 02:54:40.067527 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.067424 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 02:54:40.067527 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.067485 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jvhtj\"" Oct 31 02:54:40.067635 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.067551 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ls2x2\"" Oct 31 02:54:40.068992 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.068974 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 02:54:40.080209 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-etc-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.080322 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-k8s-cni-cncf-io\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.080322 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-konnectivity-ca\") pod \"konnectivity-agent-khwpz\" (UID: \"24d5a9bc-29a3-42b5-bf90-9468b578b0f7\") " pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:54:40.080322 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-slash\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.080322 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080290 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-etc-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.080505 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080345 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.080505 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080378 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-hostroot\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.080505 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-kubernetes\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.080505 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.080505 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2be3a160-6ada-4e51-a6bf-19745a8670f4-host\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:40.080505 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-log-socket\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.080505 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-system-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.080505 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-conf\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-cni-binary-copy\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sskj6\" (UniqueName: \"kubernetes.io/projected/af399405-2475-49b9-9fdd-7f43dff044b6-kube-api-access-sskj6\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-netd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-script-lib\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-tuned\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58487\" (UniqueName: \"kubernetes.io/projected/0facc8a8-af14-4207-bb9f-2e36129b0343-kube-api-access-58487\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-system-cni-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.080866 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080350 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-slash\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.081375 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.080909 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-konnectivity-ca\") pod \"konnectivity-agent-khwpz\" (UID: \"24d5a9bc-29a3-42b5-bf90-9468b578b0f7\") " pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:54:40.081543 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081505 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.081689 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081569 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-netd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.081689 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081588 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.081689 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081612 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2be3a160-6ada-4e51-a6bf-19745a8670f4-host\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:40.081689 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-sys-fs\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.081689 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081654 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.081689 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-systemd-units\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.082001 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081693 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-log-socket\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.082001 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-systemd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.082001 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081751 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-systemd-units\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.082001 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.082001 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081790 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-systemd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.082001 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-env-overrides\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.082001 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-netns\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.082001 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081896 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-sys-fs\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.082001 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysconfig\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.082001 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.081984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082007 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fd7955b-02fe-411e-83a4-abac6fdbb242-host-slash\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082072 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-script-lib\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082104 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-run\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082097 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fd7955b-02fe-411e-83a4-abac6fdbb242-host-slash\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-cnibin\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-dbus\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082231 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-cnibin\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-env-overrides\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee641d86-9b04-4f02-a22b-7cc55694a018-hosts-file\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-modprobe-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082352 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-system-cni-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2be3a160-6ada-4e51-a6bf-19745a8670f4-serviceca\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082434 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.082595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082504 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-config\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082599 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-dbus\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-bin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082649 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-conf-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.082698 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-registration-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.082773 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret podName:3b7047af-e1c4-4e1c-81ea-f777c15b0d97 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:40.582757243 +0000 UTC m=+3.019419251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret") pod "global-pull-secret-syncer-vtlr6" (UID: "3b7047af-e1c4-4e1c-81ea-f777c15b0d97") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-netns\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-registration-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-multus\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082879 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2be3a160-6ada-4e51-a6bf-19745a8670f4-serviceca\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082901 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-netns\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-etc-kubernetes\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.082980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee641d86-9b04-4f02-a22b-7cc55694a018-tmp-dir\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:40.083374 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.083132 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-device-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.083186 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs podName:24901959-cd66-4412-a070-aeb1cfaa8444 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:40.583169704 +0000 UTC m=+3.019831723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs") pod "network-metrics-daemon-n8slw" (UID: "24901959-cd66-4412-a070-aeb1cfaa8444") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083193 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-config\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxm7m\" (UniqueName: \"kubernetes.io/projected/2be3a160-6ada-4e51-a6bf-19745a8670f4-kube-api-access-jxm7m\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083314 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-device-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-ovn\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-node-log\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-node-log\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-cnibin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083531 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-ovn\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-multus-daemon-config\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083982 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 02:54:40.084420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.083608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-agent-certs\") pod \"konnectivity-agent-khwpz\" (UID: \"24d5a9bc-29a3-42b5-bf90-9468b578b0f7\") " pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-multus-certs\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l65r\" (UniqueName: \"kubernetes.io/projected/79426555-723a-431c-862c-8f936334be71-kube-api-access-4l65r\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-var-lib-kubelet\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-host\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-kubelet\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-bin\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovn-node-metrics-cert\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084700 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-bin\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jwx7\" (UniqueName: \"kubernetes.io/projected/ee641d86-9b04-4f02-a22b-7cc55694a018-kube-api-access-9jwx7\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-sys\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-lib-modules\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-tmp\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4vt\" (UniqueName: \"kubernetes.io/projected/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-kube-api-access-lx4vt\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-kubelet\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.085278 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084966 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fd7955b-02fe-411e-83a4-abac6fdbb242-iptables-alerter-script\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-kubelet-config\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085020 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-socket-dir-parent\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-systemd\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-249fn\" (UniqueName: \"kubernetes.io/projected/4249fb31-38de-4684-abd0-6d49c6dacd59-kube-api-access-249fn\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pnv\" (UniqueName: \"kubernetes.io/projected/24901959-cd66-4412-a070-aeb1cfaa8444-kube-api-access-g9pnv\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-socket-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdx4z\" (UniqueName: \"kubernetes.io/projected/6fd7955b-02fe-411e-83a4-abac6fdbb242-kube-api-access-zdx4z\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.084674 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-kubelet\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-var-lib-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085277 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-var-lib-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085378 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-kubelet-config\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085582 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085888 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-socket-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085940 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fd7955b-02fe-411e-83a4-abac6fdbb242-iptables-alerter-script\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:40.086296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.085980 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.088982 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.088957 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-agent-certs\") pod \"konnectivity-agent-khwpz\" (UID: \"24d5a9bc-29a3-42b5-bf90-9468b578b0f7\") " pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:54:40.091262 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.091237 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovn-node-metrics-cert\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.092667 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.092211 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:40.092667 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.092237 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:40.092667 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.092252 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vnm92 for pod openshift-network-diagnostics/network-check-target-kjmq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:40.092667 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.092309 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92 podName:e02f7dec-7aaf-4418-b520-74ce9c2ebbf0 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:40.592291325 +0000 UTC m=+3.028953350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vnm92" (UniqueName: "kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92") pod "network-check-target-kjmq9" (UID: "e02f7dec-7aaf-4418-b520-74ce9c2ebbf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:40.096762 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.096699 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sskj6\" (UniqueName: \"kubernetes.io/projected/af399405-2475-49b9-9fdd-7f43dff044b6-kube-api-access-sskj6\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.106262 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.105175 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxm7m\" (UniqueName: \"kubernetes.io/projected/2be3a160-6ada-4e51-a6bf-19745a8670f4-kube-api-access-jxm7m\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:40.107517 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.107469 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" event={"ID":"da9a871c29174a296dbd1f4a73e3997d","Type":"ContainerStarted","Data":"5e4788a9e9512eb0a3edd7845396e8b3882e48d761abb29316c78bd9543d9527"} Oct 31 02:54:40.109779 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.109754 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9pnv\" (UniqueName: \"kubernetes.io/projected/24901959-cd66-4412-a070-aeb1cfaa8444-kube-api-access-g9pnv\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:40.110082 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.110062 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdx4z\" (UniqueName: \"kubernetes.io/projected/6fd7955b-02fe-411e-83a4-abac6fdbb242-kube-api-access-zdx4z\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:40.110308 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.110286 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" event={"ID":"b7f3d03d6994989fb7ae18d976e5b40a","Type":"ContainerStarted","Data":"fcd97591b2f336f4047f85efd77085896f09401c9144412995da9b3073c7a659"} Oct 31 02:54:40.111265 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.111220 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-249fn\" (UniqueName: \"kubernetes.io/projected/4249fb31-38de-4684-abd0-6d49c6dacd59-kube-api-access-249fn\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.114054 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.114035 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4vt\" (UniqueName: \"kubernetes.io/projected/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-kube-api-access-lx4vt\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.185673 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-cni-binary-copy\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.185673 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-tuned\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.185902 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58487\" (UniqueName: \"kubernetes.io/projected/0facc8a8-af14-4207-bb9f-2e36129b0343-kube-api-access-58487\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.185902 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-netns\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.185902 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysconfig\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.185902 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-run\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.185902 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee641d86-9b04-4f02-a22b-7cc55694a018-hosts-file\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:40.185902 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185837 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-modprobe-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.185902 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.185902 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-bin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-conf-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-multus\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.185977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-etc-kubernetes\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee641d86-9b04-4f02-a22b-7cc55694a018-tmp-dir\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-cnibin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-multus-daemon-config\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-multus-certs\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l65r\" (UniqueName: \"kubernetes.io/projected/79426555-723a-431c-862c-8f936334be71-kube-api-access-4l65r\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-var-lib-kubelet\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-host\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwx7\" (UniqueName: \"kubernetes.io/projected/ee641d86-9b04-4f02-a22b-7cc55694a018-kube-api-access-9jwx7\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-sys\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.186302 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-lib-modules\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.186855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-tmp\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.186855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-kubelet\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-socket-dir-parent\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-systemd\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.186855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-k8s-cni-cncf-io\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186552 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186610 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-cnibin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.186855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186776 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee641d86-9b04-4f02-a22b-7cc55694a018-tmp-dir\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:40.186855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-hostroot\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.187342 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-kubernetes\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.187342 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-system-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.187342 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.186948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-conf\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.187342 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187163 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-conf\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.187342 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187170 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-multus-daemon-config\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.187342 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187238 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-multus-certs\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.187632 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-var-lib-kubelet\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.187632 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187539 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-host\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.187632 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187585 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-sys\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.187778 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187672 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-lib-modules\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.187829 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187783 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-socket-dir-parent\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.187829 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-netns\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.187929 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187830 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-kubelet\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.187929 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysconfig\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.187929 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187907 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-systemd\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.187929 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187911 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-run\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.188136 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187952 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-multus\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.188136 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.187957 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-etc-kubernetes\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.188136 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.188013 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee641d86-9b04-4f02-a22b-7cc55694a018-hosts-file\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:40.188136 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.188051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-kubernetes\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.188136 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.188124 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-modprobe-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.188328 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.188160 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-cni-binary-copy\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.188328 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.188176 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-system-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.188328 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.188222 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-k8s-cni-cncf-io\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.188328 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.188256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-conf-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.188328 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.188262 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-bin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.188328 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.188309 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-hostroot\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.188621 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.188440 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.190577 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.190521 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-tuned\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.190978 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.190955 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-tmp\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.199726 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.199683 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58487\" (UniqueName: \"kubernetes.io/projected/0facc8a8-af14-4207-bb9f-2e36129b0343-kube-api-access-58487\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.200262 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.200217 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l65r\" (UniqueName: \"kubernetes.io/projected/79426555-723a-431c-862c-8f936334be71-kube-api-access-4l65r\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.200588 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.200569 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwx7\" (UniqueName: \"kubernetes.io/projected/ee641d86-9b04-4f02-a22b-7cc55694a018-kube-api-access-9jwx7\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:40.290590 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.290554 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:54:40.301948 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.301926 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:54:40.302523 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:40.302496 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be3a160_6ada_4e51_a6bf_19745a8670f4.slice/crio-e274af9265f8b19a46910c59ab651f6be4819bd7b5cd43f95f590fb16d9802a8 WatchSource:0}: Error finding container e274af9265f8b19a46910c59ab651f6be4819bd7b5cd43f95f590fb16d9802a8: Status 404 returned error can't find the container with id e274af9265f8b19a46910c59ab651f6be4819bd7b5cd43f95f590fb16d9802a8 Oct 31 02:54:40.308510 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.308427 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:54:40.311226 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:40.311197 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05a3117_b24e_4810_9a49_2dd3cc9cc88b.slice/crio-1078d9d7bd59fac10155569561010a84b87815f351269d5b1bdb5778c60f17ed WatchSource:0}: Error finding container 1078d9d7bd59fac10155569561010a84b87815f351269d5b1bdb5778c60f17ed: Status 404 returned error can't find the container with id 1078d9d7bd59fac10155569561010a84b87815f351269d5b1bdb5778c60f17ed Oct 31 02:54:40.311891 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.311871 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:54:40.316717 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.316694 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:54:40.318024 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:40.317997 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd7955b_02fe_411e_83a4_abac6fdbb242.slice/crio-6e16567e212d6b5b6fc2aaff2bbfed309f2661fa76034f5faa4436a6647119da WatchSource:0}: Error finding container 6e16567e212d6b5b6fc2aaff2bbfed309f2661fa76034f5faa4436a6647119da: Status 404 returned error can't find the container with id 6e16567e212d6b5b6fc2aaff2bbfed309f2661fa76034f5faa4436a6647119da Oct 31 02:54:40.319130 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.319110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:54:40.323639 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:40.323602 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf399405_2475_49b9_9fdd_7f43dff044b6.slice/crio-e3fdb9bb2fdcd79ea89101783c588df63c453e920ffdd19b55380b9442e12616 WatchSource:0}: Error finding container e3fdb9bb2fdcd79ea89101783c588df63c453e920ffdd19b55380b9442e12616: Status 404 returned error can't find the container with id e3fdb9bb2fdcd79ea89101783c588df63c453e920ffdd19b55380b9442e12616 Oct 31 02:54:40.327127 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.327094 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:54:40.329231 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:40.329212 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4249fb31_38de_4684_abd0_6d49c6dacd59.slice/crio-0e6ed5c3012446197ef34ac0965c1fd24d0f507a35f16465365634c1c2b3a9ec WatchSource:0}: Error finding container 0e6ed5c3012446197ef34ac0965c1fd24d0f507a35f16465365634c1c2b3a9ec: Status 404 returned error can't find the container with id 0e6ed5c3012446197ef34ac0965c1fd24d0f507a35f16465365634c1c2b3a9ec Oct 31 02:54:40.331638 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:40.331612 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d5a9bc_29a3_42b5_bf90_9468b578b0f7.slice/crio-08000406ce92b78184d10f9abae51dc78c7143d0e3474535035b4776e5389ee7 WatchSource:0}: Error finding container 08000406ce92b78184d10f9abae51dc78c7143d0e3474535035b4776e5389ee7: Status 404 returned error can't find the container with id 08000406ce92b78184d10f9abae51dc78c7143d0e3474535035b4776e5389ee7 Oct 31 02:54:40.336460 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.336425 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:54:40.337800 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:40.337779 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee641d86_9b04_4f02_a22b_7cc55694a018.slice/crio-7b871bf82f5b0a5e62f3c2314e1584afcd81fb0526b6002912d5395d66c0721f WatchSource:0}: Error finding container 7b871bf82f5b0a5e62f3c2314e1584afcd81fb0526b6002912d5395d66c0721f: Status 404 returned error can't find the container with id 7b871bf82f5b0a5e62f3c2314e1584afcd81fb0526b6002912d5395d66c0721f Oct 31 02:54:40.343631 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.343602 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qhcg9" Oct 31 02:54:40.346933 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:40.346905 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0facc8a8_af14_4207_bb9f_2e36129b0343.slice/crio-65e0db778201c7d650e6c64dd222d1cf7860002a13d503134631b3abfa6d569c WatchSource:0}: Error finding container 65e0db778201c7d650e6c64dd222d1cf7860002a13d503134631b3abfa6d569c: Status 404 returned error can't find the container with id 65e0db778201c7d650e6c64dd222d1cf7860002a13d503134631b3abfa6d569c Oct 31 02:54:40.367100 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:54:40.366425 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79426555_723a_431c_862c_8f936334be71.slice/crio-7e7da63f88a8d647f954c00207e126de0c9d7078bd811b7513e53c95899263c5 WatchSource:0}: Error finding container 7e7da63f88a8d647f954c00207e126de0c9d7078bd811b7513e53c95899263c5: Status 404 returned error can't find the container with id 7e7da63f88a8d647f954c00207e126de0c9d7078bd811b7513e53c95899263c5 Oct 31 02:54:40.454451 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.454379 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:40.590654 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.590124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:40.590654 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.590231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:40.590654 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.590278 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:40.590654 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.590325 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:40.590654 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.590360 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs podName:24901959-cd66-4412-a070-aeb1cfaa8444 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:41.590339526 +0000 UTC m=+4.027001545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs") pod "network-metrics-daemon-n8slw" (UID: "24901959-cd66-4412-a070-aeb1cfaa8444") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:40.590654 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.590381 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret podName:3b7047af-e1c4-4e1c-81ea-f777c15b0d97 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:41.590371046 +0000 UTC m=+4.027033055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret") pod "global-pull-secret-syncer-vtlr6" (UID: "3b7047af-e1c4-4e1c-81ea-f777c15b0d97") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:40.690704 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:40.690668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:40.690884 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.690849 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:40.690884 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.690867 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:40.690884 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.690881 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vnm92 for pod openshift-network-diagnostics/network-check-target-kjmq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:40.691012 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:40.690947 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92 podName:e02f7dec-7aaf-4418-b520-74ce9c2ebbf0 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:41.690931946 +0000 UTC m=+4.127593964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vnm92" (UniqueName: "kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92") pod "network-check-target-kjmq9" (UID: "e02f7dec-7aaf-4418-b520-74ce9c2ebbf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:41.022710 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.022591 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 02:49:40 +0000 UTC" deadline="2028-07-31 05:23:02.569129718 +0000 UTC" Oct 31 02:54:41.022710 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.022652 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15338h28m21.546482568s" Oct 31 02:54:41.098908 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.098396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:41.098908 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:41.098531 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:54:41.131822 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.131781 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qhcg9" event={"ID":"79426555-723a-431c-862c-8f936334be71","Type":"ContainerStarted","Data":"7e7da63f88a8d647f954c00207e126de0c9d7078bd811b7513e53c95899263c5"} Oct 31 02:54:41.135310 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.135280 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" event={"ID":"0facc8a8-af14-4207-bb9f-2e36129b0343","Type":"ContainerStarted","Data":"65e0db778201c7d650e6c64dd222d1cf7860002a13d503134631b3abfa6d569c"} Oct 31 02:54:41.137595 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.137569 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mxcvd" event={"ID":"ee641d86-9b04-4f02-a22b-7cc55694a018","Type":"ContainerStarted","Data":"7b871bf82f5b0a5e62f3c2314e1584afcd81fb0526b6002912d5395d66c0721f"} Oct 31 02:54:41.139487 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.139460 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" event={"ID":"4249fb31-38de-4684-abd0-6d49c6dacd59","Type":"ContainerStarted","Data":"0e6ed5c3012446197ef34ac0965c1fd24d0f507a35f16465365634c1c2b3a9ec"} Oct 31 02:54:41.144004 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.143977 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tq4b7" event={"ID":"6fd7955b-02fe-411e-83a4-abac6fdbb242","Type":"ContainerStarted","Data":"6e16567e212d6b5b6fc2aaff2bbfed309f2661fa76034f5faa4436a6647119da"} Oct 31 02:54:41.153111 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.153081 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" event={"ID":"c05a3117-b24e-4810-9a49-2dd3cc9cc88b","Type":"ContainerStarted","Data":"1078d9d7bd59fac10155569561010a84b87815f351269d5b1bdb5778c60f17ed"} Oct 31 02:54:41.156579 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.156555 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-khwpz" event={"ID":"24d5a9bc-29a3-42b5-bf90-9468b578b0f7","Type":"ContainerStarted","Data":"08000406ce92b78184d10f9abae51dc78c7143d0e3474535035b4776e5389ee7"} Oct 31 02:54:41.169402 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.169376 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" event={"ID":"af399405-2475-49b9-9fdd-7f43dff044b6","Type":"ContainerStarted","Data":"e3fdb9bb2fdcd79ea89101783c588df63c453e920ffdd19b55380b9442e12616"} Oct 31 02:54:41.178245 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.178217 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xlsrl" event={"ID":"2be3a160-6ada-4e51-a6bf-19745a8670f4","Type":"ContainerStarted","Data":"e274af9265f8b19a46910c59ab651f6be4819bd7b5cd43f95f590fb16d9802a8"} Oct 31 02:54:41.597886 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.597072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:41.597886 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.597130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:41.597886 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:41.597277 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:41.597886 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:41.597344 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs podName:24901959-cd66-4412-a070-aeb1cfaa8444 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:43.59732284 +0000 UTC m=+6.033984849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs") pod "network-metrics-daemon-n8slw" (UID: "24901959-cd66-4412-a070-aeb1cfaa8444") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:41.597886 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:41.597747 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:41.597886 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:41.597799 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret podName:3b7047af-e1c4-4e1c-81ea-f777c15b0d97 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:43.597784696 +0000 UTC m=+6.034446699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret") pod "global-pull-secret-syncer-vtlr6" (UID: "3b7047af-e1c4-4e1c-81ea-f777c15b0d97") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:41.697787 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:41.697730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:41.697968 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:41.697944 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:41.697968 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:41.697966 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:41.698094 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:41.697979 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vnm92 for pod openshift-network-diagnostics/network-check-target-kjmq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:41.698094 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:41.698038 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92 podName:e02f7dec-7aaf-4418-b520-74ce9c2ebbf0 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:43.698020946 +0000 UTC m=+6.134682954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vnm92" (UniqueName: "kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92") pod "network-check-target-kjmq9" (UID: "e02f7dec-7aaf-4418-b520-74ce9c2ebbf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:42.023349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:42.023221 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 02:49:40 +0000 UTC" deadline="2028-06-02 19:22:04.655482645 +0000 UTC" Oct 31 02:54:42.023349 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:42.023265 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13936h27m22.63222313s" Oct 31 02:54:42.100251 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:42.100215 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:42.100408 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:42.100369 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:54:42.100913 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:42.100887 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:42.101030 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:42.100996 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:54:42.196243 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:42.196207 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" event={"ID":"da9a871c29174a296dbd1f4a73e3997d","Type":"ContainerStarted","Data":"445cdd83fb475b12c227e36fdfc29b1b747f10071e40246e7221dc0494687c1b"} Oct 31 02:54:43.099171 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:43.098666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:43.099171 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:43.098802 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:54:43.615835 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:43.614642 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:43.615835 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:43.614975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:43.615835 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:43.615135 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:43.615835 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:43.615203 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret podName:3b7047af-e1c4-4e1c-81ea-f777c15b0d97 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:47.615184854 +0000 UTC m=+10.051846871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret") pod "global-pull-secret-syncer-vtlr6" (UID: "3b7047af-e1c4-4e1c-81ea-f777c15b0d97") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:43.615835 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:43.615652 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:43.615835 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:43.615705 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs podName:24901959-cd66-4412-a070-aeb1cfaa8444 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:47.615688766 +0000 UTC m=+10.052350785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs") pod "network-metrics-daemon-n8slw" (UID: "24901959-cd66-4412-a070-aeb1cfaa8444") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:43.715939 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:43.715902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:43.716122 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:43.716088 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:43.716122 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:43.716107 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:43.716122 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:43.716119 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vnm92 for pod openshift-network-diagnostics/network-check-target-kjmq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:43.716282 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:43.716173 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92 podName:e02f7dec-7aaf-4418-b520-74ce9c2ebbf0 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:47.716156632 +0000 UTC m=+10.152818640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vnm92" (UniqueName: "kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92") pod "network-check-target-kjmq9" (UID: "e02f7dec-7aaf-4418-b520-74ce9c2ebbf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:44.099459 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:44.099159 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:44.099459 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:44.099300 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:54:44.099948 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:44.099550 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:44.099948 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:44.099709 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:54:44.211064 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:44.210963 2577 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="cdcf74d2f00011601d074c2e4b8df0eb6c6d1fd13ecea8c7cbcc01c2bf9d5a3b" exitCode=0 Oct 31 02:54:44.211235 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:44.211084 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" event={"ID":"4249fb31-38de-4684-abd0-6d49c6dacd59","Type":"ContainerDied","Data":"cdcf74d2f00011601d074c2e4b8df0eb6c6d1fd13ecea8c7cbcc01c2bf9d5a3b"} Oct 31 02:54:44.222533 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:44.222499 2577 generic.go:356] "Generic (PLEG): container finished" podID="b7f3d03d6994989fb7ae18d976e5b40a" containerID="6def1099e5940d5b3d545f521e61bd9567042d4ef0fb0a05d9a91c544958b404" exitCode=0 Oct 31 02:54:44.222644 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:44.222542 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" event={"ID":"b7f3d03d6994989fb7ae18d976e5b40a","Type":"ContainerDied","Data":"6def1099e5940d5b3d545f521e61bd9567042d4ef0fb0a05d9a91c544958b404"} Oct 31 02:54:44.235853 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:44.235708 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" podStartSLOduration=5.235690421 podStartE2EDuration="5.235690421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:42.212197197 +0000 UTC m=+4.648859224" watchObservedRunningTime="2026-10-31 02:54:44.235690421 +0000 UTC m=+6.672352449" Oct 31 02:54:45.098841 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:45.098809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:45.098973 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:45.098939 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:54:45.239328 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:45.239254 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" event={"ID":"b7f3d03d6994989fb7ae18d976e5b40a","Type":"ContainerStarted","Data":"350d706d0dfd2eee4253f84da9d407a706f93c1c2117bb5a8c31bfc6b58bfbc1"} Oct 31 02:54:45.247772 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:45.247740 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-khwpz" event={"ID":"24d5a9bc-29a3-42b5-bf90-9468b578b0f7","Type":"ContainerStarted","Data":"b5db8354c69d52a8a597538b6b37ef31f78a3f3c5dd3ec03229e71b4f4f8e15d"} Oct 31 02:54:45.255452 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:45.255404 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" podStartSLOduration=6.255390991 podStartE2EDuration="6.255390991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:45.255177306 +0000 UTC m=+7.691839331" watchObservedRunningTime="2026-10-31 02:54:45.255390991 +0000 UTC m=+7.692053009" Oct 31 02:54:46.099186 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:46.099153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:46.099369 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:46.099323 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:54:46.099891 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:46.099869 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:46.100022 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:46.099997 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:54:46.255976 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:46.255461 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xlsrl" event={"ID":"2be3a160-6ada-4e51-a6bf-19745a8670f4","Type":"ContainerStarted","Data":"6e6ffed1e1853968ee74251af3a4be66898662344aa4d4b6cc5cd3dc0bea0631"} Oct 31 02:54:46.283075 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:46.282904 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-khwpz" podStartSLOduration=3.640148205 podStartE2EDuration="8.282885784s" totalImagesPullingTime="4.642737579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:45.271342399 +0000 UTC m=+7.708004424" watchObservedRunningTime="2026-10-31 02:54:46.282885784 +0000 UTC m=+8.719547810" Oct 31 02:54:46.283227 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:46.283087 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xlsrl" podStartSLOduration=2.903723809 podStartE2EDuration="8.283076149s" totalImagesPullingTime="5.37935234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:46.281437814 +0000 UTC m=+8.718099839" watchObservedRunningTime="2026-10-31 02:54:46.283076149 +0000 UTC m=+8.719738174" Oct 31 02:54:47.098638 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:47.098559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:47.098828 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:47.098692 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:54:47.259523 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:47.259488 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" event={"ID":"af399405-2475-49b9-9fdd-7f43dff044b6","Type":"ContainerStarted","Data":"c49c76a3d94222371753095ee699448f649a498e6c20964fde3080895f25a04b"} Oct 31 02:54:47.649437 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:47.649401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:47.649626 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:47.649447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:47.649626 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:47.649564 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:47.649626 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:47.649565 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:47.649787 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:47.649638 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs podName:24901959-cd66-4412-a070-aeb1cfaa8444 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:55.649616838 +0000 UTC m=+18.086278853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs") pod "network-metrics-daemon-n8slw" (UID: "24901959-cd66-4412-a070-aeb1cfaa8444") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:47.649787 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:47.649657 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret podName:3b7047af-e1c4-4e1c-81ea-f777c15b0d97 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:55.649647106 +0000 UTC m=+18.086309108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret") pod "global-pull-secret-syncer-vtlr6" (UID: "3b7047af-e1c4-4e1c-81ea-f777c15b0d97") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:47.750329 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:47.750295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:47.750507 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:47.750492 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:47.750564 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:47.750516 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:47.750564 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:47.750528 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vnm92 for pod openshift-network-diagnostics/network-check-target-kjmq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:47.750646 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:47.750587 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92 podName:e02f7dec-7aaf-4418-b520-74ce9c2ebbf0 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:55.750567501 +0000 UTC m=+18.187229525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vnm92" (UniqueName: "kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92") pod "network-check-target-kjmq9" (UID: "e02f7dec-7aaf-4418-b520-74ce9c2ebbf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:48.100674 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:48.100265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:48.100674 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:48.100372 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:54:48.101118 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:48.100966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:48.101118 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:48.101081 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:54:48.267130 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:48.266551 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qhcg9" event={"ID":"79426555-723a-431c-862c-8f936334be71","Type":"ContainerStarted","Data":"b2fb90d2e8e902b4e739351bb8b02493465c6f72ff485e4a68e40e67bf2f8475"} Oct 31 02:54:48.402849 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:48.402810 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 02:54:49.053371 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.052807 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T02:54:48.40283136Z","UUID":"b9045b13-3c92-4542-b3c5-a48eb202d914","Handler":null,"Name":"","Endpoint":""} Oct 31 02:54:49.054454 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.054146 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 02:54:49.054454 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.054177 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 02:54:49.098312 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.098280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:49.098534 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:49.098401 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:54:49.270711 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.270629 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mxcvd" event={"ID":"ee641d86-9b04-4f02-a22b-7cc55694a018","Type":"ContainerStarted","Data":"38ff3ebfe3a1c3060fc6dc9a0c9c5cd59637447ca576f58a5d4c194a0d0af940"} Oct 31 02:54:49.273890 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.273861 2577 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="55ee6ab30963ac6809387781778e51e53970990f0dd0270e029dae47a2ef83e5" exitCode=0 Oct 31 02:54:49.274496 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.274466 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" event={"ID":"4249fb31-38de-4684-abd0-6d49c6dacd59","Type":"ContainerDied","Data":"55ee6ab30963ac6809387781778e51e53970990f0dd0270e029dae47a2ef83e5"} Oct 31 02:54:49.277780 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.277750 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" event={"ID":"af399405-2475-49b9-9fdd-7f43dff044b6","Type":"ContainerStarted","Data":"29fb76d41b89df05367d5830af7b0fdb7b0e6ba2367f8af3fc1a5ca4535b7728"} Oct 31 02:54:49.287256 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.287213 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-mxcvd" podStartSLOduration=2.753491515 podStartE2EDuration="11.287198216s" totalImagesPullingTime="8.533706701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:49.286589987 +0000 UTC m=+11.723252014" watchObservedRunningTime="2026-10-31 02:54:49.287198216 +0000 UTC m=+11.723860244" Oct 31 02:54:49.287403 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.287380 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qhcg9" podStartSLOduration=3.637470053 podStartE2EDuration="11.287373156s" totalImagesPullingTime="7.649903103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:48.290900044 +0000 UTC m=+10.727562068" watchObservedRunningTime="2026-10-31 02:54:49.287373156 +0000 UTC m=+11.724035180" Oct 31 02:54:49.981063 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.980778 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:54:49.981506 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:49.981482 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:54:50.098479 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:50.098255 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:50.098636 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:50.098602 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:54:50.099206 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:50.099183 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:50.099338 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:50.099313 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:54:50.283050 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:50.283000 2577 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="408219afa6fab4174ea6bf82454d3074329c76e00c2398e7fefba42306eb8653" exitCode=0 Oct 31 02:54:50.283488 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:50.283057 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" event={"ID":"4249fb31-38de-4684-abd0-6d49c6dacd59","Type":"ContainerDied","Data":"408219afa6fab4174ea6bf82454d3074329c76e00c2398e7fefba42306eb8653"} Oct 31 02:54:50.285131 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:50.285094 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tq4b7" event={"ID":"6fd7955b-02fe-411e-83a4-abac6fdbb242","Type":"ContainerStarted","Data":"58a0c509ce0fc07c7ca838325ddf670ebc848f8416f2faa65064a39b05db38b5"} Oct 31 02:54:50.289802 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:50.289777 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" event={"ID":"af399405-2475-49b9-9fdd-7f43dff044b6","Type":"ContainerStarted","Data":"2ed52b2a37068b1c1d8222611d299d68d7bc4f3fc4068d9e00bceab0871cc1c5"} Oct 31 02:54:50.327749 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:50.327687 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" podStartSLOduration=3.662924357 podStartE2EDuration="12.32767046s" totalImagesPullingTime="8.664746103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:50.327286874 +0000 UTC m=+12.763948899" watchObservedRunningTime="2026-10-31 02:54:50.32767046 +0000 UTC m=+12.764332484" Oct 31 02:54:50.346118 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:50.346070 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tq4b7" podStartSLOduration=4.321042451 podStartE2EDuration="12.34605405s" totalImagesPullingTime="8.025011599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:50.345539412 +0000 UTC m=+12.782201438" watchObservedRunningTime="2026-10-31 02:54:50.34605405 +0000 UTC m=+12.782716065" Oct 31 02:54:51.098923 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:51.098847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:51.099110 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:51.098957 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:54:51.292592 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:51.292546 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" event={"ID":"0facc8a8-af14-4207-bb9f-2e36129b0343","Type":"ContainerStarted","Data":"8bd96658666de259caa5190cd564332aa0252ce7a045eeb41854050e25ef20a3"} Oct 31 02:54:52.101361 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:52.101293 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:52.101510 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:52.101437 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:54:52.101510 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:52.101494 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:52.101626 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:52.101577 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:54:52.297619 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:52.297580 2577 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="be4e08ec20d65fe748e1c466007bdec55bcd5f0edae7a9750d17ea72b71976e2" exitCode=0 Oct 31 02:54:52.298285 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:52.297699 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" event={"ID":"4249fb31-38de-4684-abd0-6d49c6dacd59","Type":"ContainerDied","Data":"be4e08ec20d65fe748e1c466007bdec55bcd5f0edae7a9750d17ea72b71976e2"} Oct 31 02:54:52.321473 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:52.321429 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" podStartSLOduration=3.937504611 podStartE2EDuration="14.321413796s" totalImagesPullingTime="10.383909185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:51.309955582 +0000 UTC m=+13.746617605" watchObservedRunningTime="2026-10-31 02:54:52.321413796 +0000 UTC m=+14.758075820" Oct 31 02:54:53.098855 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:53.098821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:53.099028 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:53.098943 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:54:54.101935 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:54.101904 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:54.102536 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:54.102059 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:54:54.102536 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:54.102078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:54.102536 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:54.102193 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:54:55.098976 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:55.098801 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:55.099138 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:55.099069 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:54:55.305790 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:55.305758 2577 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="d1989f01e4d618ef3c28dd8e122b1ad910b2a7eb88874a0b677df6a2bc2f3644" exitCode=0 Oct 31 02:54:55.306158 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:55.305804 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" event={"ID":"4249fb31-38de-4684-abd0-6d49c6dacd59","Type":"ContainerDied","Data":"d1989f01e4d618ef3c28dd8e122b1ad910b2a7eb88874a0b677df6a2bc2f3644"} Oct 31 02:54:55.723392 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:55.723367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:55.723522 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:55.723400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:55.723522 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:55.723488 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:55.723582 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:55.723529 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:55.723582 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:55.723538 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs podName:24901959-cd66-4412-a070-aeb1cfaa8444 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:11.723523298 +0000 UTC m=+34.160185301 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs") pod "network-metrics-daemon-n8slw" (UID: "24901959-cd66-4412-a070-aeb1cfaa8444") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:55.723644 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:55.723605 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret podName:3b7047af-e1c4-4e1c-81ea-f777c15b0d97 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:11.723586359 +0000 UTC m=+34.160248370 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret") pod "global-pull-secret-syncer-vtlr6" (UID: "3b7047af-e1c4-4e1c-81ea-f777c15b0d97") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:55.824275 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:55.824247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:55.824407 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:55.824390 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:55.824407 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:55.824407 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:55.824483 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:55.824415 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vnm92 for pod openshift-network-diagnostics/network-check-target-kjmq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:55.824483 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:55.824463 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92 podName:e02f7dec-7aaf-4418-b520-74ce9c2ebbf0 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:11.824447459 +0000 UTC m=+34.261109461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vnm92" (UniqueName: "kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92") pod "network-check-target-kjmq9" (UID: "e02f7dec-7aaf-4418-b520-74ce9c2ebbf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:56.101790 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:56.101770 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:56.101921 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:56.101898 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:54:56.101965 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:56.101940 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:56.102019 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:56.102005 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:54:56.311022 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:56.310986 2577 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="64f811666cfc70e12368df1bd481031a41a97a8e5f9d4a0c9a1ef829efc2905e" exitCode=0 Oct 31 02:54:56.311559 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:56.311069 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" event={"ID":"4249fb31-38de-4684-abd0-6d49c6dacd59","Type":"ContainerDied","Data":"64f811666cfc70e12368df1bd481031a41a97a8e5f9d4a0c9a1ef829efc2905e"} Oct 31 02:54:56.314143 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:56.314126 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" event={"ID":"c05a3117-b24e-4810-9a49-2dd3cc9cc88b","Type":"ContainerStarted","Data":"57462e64612744227cefb8a56cef83dc81d61835161097ab6fd7030594d031de"} Oct 31 02:54:56.314212 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:56.314197 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" event={"ID":"c05a3117-b24e-4810-9a49-2dd3cc9cc88b","Type":"ContainerStarted","Data":"ceffb727c4d2ede6bbc87cd5c8f3d44c290c8179875b61d4a6562e922d424d10"} Oct 31 02:54:56.314249 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:56.314239 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" event={"ID":"c05a3117-b24e-4810-9a49-2dd3cc9cc88b","Type":"ContainerStarted","Data":"a06192010dec36bfa88f14456ab7da1e785147e6b437f8f272384c312c618632"} Oct 31 02:54:56.314321 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:56.314271 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" event={"ID":"c05a3117-b24e-4810-9a49-2dd3cc9cc88b","Type":"ContainerStarted","Data":"11d68ed3739828674f85b39a261ac9a8635bd36c57261330df2abc37b7ce02d6"} Oct 31 02:54:56.314354 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:56.314343 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" event={"ID":"c05a3117-b24e-4810-9a49-2dd3cc9cc88b","Type":"ContainerStarted","Data":"a58fccb263f90b3c7499dacd177ec160e8cf2900af0f467c5adf442110a877e2"} Oct 31 02:54:56.314381 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:56.314370 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" event={"ID":"c05a3117-b24e-4810-9a49-2dd3cc9cc88b","Type":"ContainerStarted","Data":"8ad18aa6ccedb1cdba6085594a5daaeb6bd42291ce6fd185321bf75a793a26d7"} Oct 31 02:54:57.098673 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:57.098640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:57.098814 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:57.098742 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:54:57.319590 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:57.319555 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" event={"ID":"4249fb31-38de-4684-abd0-6d49c6dacd59","Type":"ContainerStarted","Data":"7183e9b611f6c9b5955f82aa33643883979a306efe16d8cc8530fbe8d7452281"} Oct 31 02:54:57.344319 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:57.344268 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g9pb4" podStartSLOduration=8.13378626 podStartE2EDuration="19.344255015s" totalImagesPullingTime="11.210468755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:57.34314496 +0000 UTC m=+19.779806983" watchObservedRunningTime="2026-10-31 02:54:57.344255015 +0000 UTC m=+19.780917038" Oct 31 02:54:58.098697 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:58.098624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:54:58.098864 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:58.098702 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:54:58.098864 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:58.098796 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:54:58.098927 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:58.098882 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:54:58.324569 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:58.324535 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" event={"ID":"c05a3117-b24e-4810-9a49-2dd3cc9cc88b","Type":"ContainerStarted","Data":"50cd8c3c03fef3610c04a50ff8eb58943bfab6e693ca6ef3fcace1db802ad6ec"} Oct 31 02:54:59.098586 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:54:59.098559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:54:59.098730 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:54:59.098657 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:55:00.098579 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:00.098504 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:00.099343 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:00.098610 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:55:00.099343 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:00.098651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:00.099343 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:00.098705 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:55:00.235535 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:00.235508 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:55:00.235706 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:00.235628 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:00.236119 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:00.236103 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:55:00.332232 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:00.332198 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" event={"ID":"c05a3117-b24e-4810-9a49-2dd3cc9cc88b","Type":"ContainerStarted","Data":"8cf255b668fad80464b79b79e5e78caac11261ff7ff7cbdaa6a6941586b50f13"} Oct 31 02:55:00.332430 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:00.332409 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:00.347012 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:00.346990 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:00.390553 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:00.390476 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" podStartSLOduration=7.343491113 podStartE2EDuration="22.390464294s" totalImagesPullingTime="15.046973181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:00.389423591 +0000 UTC m=+22.826085638" watchObservedRunningTime="2026-10-31 02:55:00.390464294 +0000 UTC m=+22.827126317" Oct 31 02:55:01.098548 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:01.098095 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:01.098548 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:01.098218 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:55:01.334270 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:01.334242 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:01.334420 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:01.334324 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:01.350319 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:01.350155 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:01.609046 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:01.608947 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vtlr6"] Oct 31 02:55:01.609187 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:01.609149 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:01.609250 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:01.609231 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:55:01.612023 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:01.611988 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kjmq9"] Oct 31 02:55:01.612166 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:01.612107 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:01.612222 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:01.612206 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:55:01.612553 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:01.612534 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n8slw"] Oct 31 02:55:01.612655 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:01.612623 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:01.612718 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:01.612702 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:55:03.098586 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:03.098556 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:03.099117 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:03.098557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:03.099117 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:03.098664 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:55:03.099117 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:03.098679 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:03.099117 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:03.098791 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:55:03.099117 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:03.098878 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:55:05.098618 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:05.098580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:05.098618 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:05.098618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:05.099023 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:05.098628 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:05.099023 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:05.098716 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:55:05.099023 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:05.098823 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:55:05.099023 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:05.098897 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:55:07.098940 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.098901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:07.098940 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.098943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:07.099498 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.099029 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:07.099498 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:07.099077 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8slw" podUID="24901959-cd66-4412-a070-aeb1cfaa8444" Oct 31 02:55:07.099498 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:07.099121 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjmq9" podUID="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" Oct 31 02:55:07.099498 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:07.099177 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vtlr6" podUID="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" Oct 31 02:55:07.892925 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.892899 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeReady" Oct 31 02:55:07.893091 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.893079 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 02:55:07.939159 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.939131 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cqgsq"] Oct 31 02:55:07.945261 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.945238 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hwt9g"] Oct 31 02:55:07.945395 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.945354 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:07.947763 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.947744 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:07.952941 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.952366 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqgsq"] Oct 31 02:55:07.962511 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.962479 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 02:55:07.962709 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.962688 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hwt9g"] Oct 31 02:55:07.962948 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.962928 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jsfq8\"" Oct 31 02:55:07.963028 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.962985 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 02:55:07.963028 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.963016 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 02:55:07.963028 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.963065 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 02:55:07.963204 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.963102 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x8dcs\"" Oct 31 02:55:07.963295 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:07.963275 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 02:55:08.105377 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.105337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/810e723f-fc3a-4477-a30d-dc086bd338b1-config-volume\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.105813 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.105389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.105813 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.105469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/810e723f-fc3a-4477-a30d-dc086bd338b1-tmp-dir\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.105813 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.105583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4ll\" (UniqueName: \"kubernetes.io/projected/b4bb32ff-188e-4471-8c62-08fae7e26c8c-kube-api-access-mv4ll\") pod \"ingress-canary-cqgsq\" (UID: \"b4bb32ff-188e-4471-8c62-08fae7e26c8c\") " pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:08.105813 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.105632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvb7x\" (UniqueName: \"kubernetes.io/projected/810e723f-fc3a-4477-a30d-dc086bd338b1-kube-api-access-jvb7x\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.105813 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.105733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4bb32ff-188e-4471-8c62-08fae7e26c8c-cert\") pod \"ingress-canary-cqgsq\" (UID: \"b4bb32ff-188e-4471-8c62-08fae7e26c8c\") " pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:08.206545 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.206513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvb7x\" (UniqueName: \"kubernetes.io/projected/810e723f-fc3a-4477-a30d-dc086bd338b1-kube-api-access-jvb7x\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.206700 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.206559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4bb32ff-188e-4471-8c62-08fae7e26c8c-cert\") pod \"ingress-canary-cqgsq\" (UID: \"b4bb32ff-188e-4471-8c62-08fae7e26c8c\") " pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:08.206700 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.206619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/810e723f-fc3a-4477-a30d-dc086bd338b1-config-volume\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.206700 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.206643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.206700 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.206664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/810e723f-fc3a-4477-a30d-dc086bd338b1-tmp-dir\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.206700 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.206695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4ll\" (UniqueName: \"kubernetes.io/projected/b4bb32ff-188e-4471-8c62-08fae7e26c8c-kube-api-access-mv4ll\") pod \"ingress-canary-cqgsq\" (UID: \"b4bb32ff-188e-4471-8c62-08fae7e26c8c\") " pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:08.206958 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:08.206810 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:08.206958 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:08.206889 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls podName:810e723f-fc3a-4477-a30d-dc086bd338b1 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:08.706867387 +0000 UTC m=+31.143529392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls") pod "dns-default-hwt9g" (UID: "810e723f-fc3a-4477-a30d-dc086bd338b1") : secret "dns-default-metrics-tls" not found Oct 31 02:55:08.207127 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.207109 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/810e723f-fc3a-4477-a30d-dc086bd338b1-tmp-dir\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.207345 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.207323 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/810e723f-fc3a-4477-a30d-dc086bd338b1-config-volume\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.210497 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.210469 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4bb32ff-188e-4471-8c62-08fae7e26c8c-cert\") pod \"ingress-canary-cqgsq\" (UID: \"b4bb32ff-188e-4471-8c62-08fae7e26c8c\") " pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:08.218644 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.218623 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4ll\" (UniqueName: \"kubernetes.io/projected/b4bb32ff-188e-4471-8c62-08fae7e26c8c-kube-api-access-mv4ll\") pod \"ingress-canary-cqgsq\" (UID: \"b4bb32ff-188e-4471-8c62-08fae7e26c8c\") " pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:08.218754 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.218621 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvb7x\" (UniqueName: \"kubernetes.io/projected/810e723f-fc3a-4477-a30d-dc086bd338b1-kube-api-access-jvb7x\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.259189 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.259166 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:08.582429 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.582346 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqgsq"] Oct 31 02:55:08.585685 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:55:08.585657 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bb32ff_188e_4471_8c62_08fae7e26c8c.slice/crio-b88322dba7979796d4eaacb871386a58c921cf9ac8b0dfe95f7f6180c7dd09aa WatchSource:0}: Error finding container b88322dba7979796d4eaacb871386a58c921cf9ac8b0dfe95f7f6180c7dd09aa: Status 404 returned error can't find the container with id b88322dba7979796d4eaacb871386a58c921cf9ac8b0dfe95f7f6180c7dd09aa Oct 31 02:55:08.710514 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:08.710484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:08.710667 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:08.710637 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:08.710712 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:08.710705 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls podName:810e723f-fc3a-4477-a30d-dc086bd338b1 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:09.71068914 +0000 UTC m=+32.147351154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls") pod "dns-default-hwt9g" (UID: "810e723f-fc3a-4477-a30d-dc086bd338b1") : secret "dns-default-metrics-tls" not found Oct 31 02:55:09.098352 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.098307 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:09.098541 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.098374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:09.098541 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.098475 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:09.101964 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.101936 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 02:55:09.102146 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.102127 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 02:55:09.102146 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.102132 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q5ns6\"" Oct 31 02:55:09.102274 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.102189 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7qc7w\"" Oct 31 02:55:09.102274 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.102196 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 02:55:09.102274 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.102253 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 02:55:09.102396 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.102320 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rkjm6\"" Oct 31 02:55:09.352013 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.351940 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqgsq" event={"ID":"b4bb32ff-188e-4471-8c62-08fae7e26c8c","Type":"ContainerStarted","Data":"b88322dba7979796d4eaacb871386a58c921cf9ac8b0dfe95f7f6180c7dd09aa"} Oct 31 02:55:09.718779 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:09.718744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:09.718915 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:09.718856 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:09.718915 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:09.718908 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls podName:810e723f-fc3a-4477-a30d-dc086bd338b1 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:11.718895181 +0000 UTC m=+34.155557183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls") pod "dns-default-hwt9g" (UID: "810e723f-fc3a-4477-a30d-dc086bd338b1") : secret "dns-default-metrics-tls" not found Oct 31 02:55:11.732075 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:11.732030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:11.732777 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:11.732082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:11.732777 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:11.732142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:11.732777 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:11.732240 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:11.732777 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:11.732260 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 02:55:11.732777 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:11.732318 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls podName:810e723f-fc3a-4477-a30d-dc086bd338b1 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:15.732296775 +0000 UTC m=+38.168958777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls") pod "dns-default-hwt9g" (UID: "810e723f-fc3a-4477-a30d-dc086bd338b1") : secret "dns-default-metrics-tls" not found Oct 31 02:55:11.732777 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:11.732338 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs podName:24901959-cd66-4412-a070-aeb1cfaa8444 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:43.732328114 +0000 UTC m=+66.168990117 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs") pod "network-metrics-daemon-n8slw" (UID: "24901959-cd66-4412-a070-aeb1cfaa8444") : secret "metrics-daemon-secret" not found Oct 31 02:55:11.734679 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:11.734655 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:11.819279 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:11.819241 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:11.832449 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:11.832416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:11.835340 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:11.835319 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:11.838018 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:11.837998 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:12.125187 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.125118 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vtlr6"] Oct 31 02:55:12.128454 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:55:12.128426 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7047af_e1c4_4e1c_81ea_f777c15b0d97.slice/crio-84d3f2282ed849c17fdcb74923acf981d206fecde9523444386bcfcbd893502e WatchSource:0}: Error finding container 84d3f2282ed849c17fdcb74923acf981d206fecde9523444386bcfcbd893502e: Status 404 returned error can't find the container with id 84d3f2282ed849c17fdcb74923acf981d206fecde9523444386bcfcbd893502e Oct 31 02:55:12.158067 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.158030 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kjmq9"] Oct 31 02:55:12.161251 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:55:12.161216 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02f7dec_7aaf_4418_b520_74ce9c2ebbf0.slice/crio-f19e9732a1bd749e2676e0d8b714829c06324110d140810616c414f393247717 WatchSource:0}: Error finding container f19e9732a1bd749e2676e0d8b714829c06324110d140810616c414f393247717: Status 404 returned error can't find the container with id f19e9732a1bd749e2676e0d8b714829c06324110d140810616c414f393247717 Oct 31 02:55:12.358985 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.358950 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kjmq9" event={"ID":"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0","Type":"ContainerStarted","Data":"f19e9732a1bd749e2676e0d8b714829c06324110d140810616c414f393247717"} Oct 31 02:55:12.360604 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.360569 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vtlr6" event={"ID":"3b7047af-e1c4-4e1c-81ea-f777c15b0d97","Type":"ContainerStarted","Data":"84d3f2282ed849c17fdcb74923acf981d206fecde9523444386bcfcbd893502e"} Oct 31 02:55:12.683479 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.683448 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rb8r5"] Oct 31 02:55:12.706296 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.706261 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rb8r5"] Oct 31 02:55:12.706458 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.706395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.744469 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.744192 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 02:55:12.744469 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.744359 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 02:55:12.744469 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.744420 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 02:55:12.762845 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.762817 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sgkvt\"" Oct 31 02:55:12.763085 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.763062 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 02:55:12.825774 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.825742 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8"] Oct 31 02:55:12.836569 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.836537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8" Oct 31 02:55:12.837306 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.837277 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8"] Oct 31 02:55:12.841496 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.841467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ded5081-b545-47f2-be60-975cf616f05f-crio-socket\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.841610 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.841557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.841610 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.841585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ded5081-b545-47f2-be60-975cf616f05f-data-volume\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.841610 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.841604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdh6m\" (UniqueName: \"kubernetes.io/projected/4ded5081-b545-47f2-be60-975cf616f05f-kube-api-access-gdh6m\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.841772 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.841658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.907955 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.907923 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hnld2\"" Oct 31 02:55:12.909999 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.909975 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 02:55:12.909999 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.909999 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:12.942657 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.942585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.942657 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.942650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ded5081-b545-47f2-be60-975cf616f05f-crio-socket\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.942844 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:12.942681 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 02:55:12.942844 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.942684 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wsb\" (UniqueName: \"kubernetes.io/projected/88c4d958-b761-4b20-b3dd-c02e87b8b10c-kube-api-access-t4wsb\") pod \"migrator-66df5d496b-xgtr8\" (UID: \"88c4d958-b761-4b20-b3dd-c02e87b8b10c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8" Oct 31 02:55:12.942844 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.942797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.942844 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.942839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ded5081-b545-47f2-be60-975cf616f05f-data-volume\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.943060 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.942866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdh6m\" (UniqueName: \"kubernetes.io/projected/4ded5081-b545-47f2-be60-975cf616f05f-kube-api-access-gdh6m\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.943060 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.942927 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ded5081-b545-47f2-be60-975cf616f05f-crio-socket\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.943060 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:12.943019 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:12.943208 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:12.943117 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.443096105 +0000 UTC m=+35.879758119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:12.943208 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:12.943171 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.443160427 +0000 UTC m=+35.879822448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : configmap "kube-rbac-proxy" not found Oct 31 02:55:12.943413 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.943394 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ded5081-b545-47f2-be60-975cf616f05f-data-volume\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:12.956644 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:12.956617 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdh6m\" (UniqueName: \"kubernetes.io/projected/4ded5081-b545-47f2-be60-975cf616f05f-kube-api-access-gdh6m\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:13.043417 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:13.043378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wsb\" (UniqueName: \"kubernetes.io/projected/88c4d958-b761-4b20-b3dd-c02e87b8b10c-kube-api-access-t4wsb\") pod \"migrator-66df5d496b-xgtr8\" (UID: \"88c4d958-b761-4b20-b3dd-c02e87b8b10c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8" Oct 31 02:55:13.061861 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:13.061539 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wsb\" (UniqueName: \"kubernetes.io/projected/88c4d958-b761-4b20-b3dd-c02e87b8b10c-kube-api-access-t4wsb\") pod \"migrator-66df5d496b-xgtr8\" (UID: \"88c4d958-b761-4b20-b3dd-c02e87b8b10c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8" Oct 31 02:55:13.160170 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:13.160126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8" Oct 31 02:55:13.446387 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:13.446356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:13.446542 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:13.446403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:13.446542 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:13.446503 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 02:55:13.446542 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:13.446522 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:13.446660 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:13.446570 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:14.446552499 +0000 UTC m=+36.883214519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : configmap "kube-rbac-proxy" not found Oct 31 02:55:13.446660 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:13.446585 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:14.446578916 +0000 UTC m=+36.883240918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:13.471055 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:13.471014 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8"] Oct 31 02:55:13.475177 ip-10-0-129-107 kubenswrapper[2577]: W1031 02:55:13.475147 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c4d958_b761_4b20_b3dd_c02e87b8b10c.slice/crio-ab637ea3ade080ca0cf898e7442782c7fb2722ff45e1d7d5d3b6d0905833cb2a WatchSource:0}: Error finding container ab637ea3ade080ca0cf898e7442782c7fb2722ff45e1d7d5d3b6d0905833cb2a: Status 404 returned error can't find the container with id ab637ea3ade080ca0cf898e7442782c7fb2722ff45e1d7d5d3b6d0905833cb2a Oct 31 02:55:14.368719 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:14.368687 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqgsq" event={"ID":"b4bb32ff-188e-4471-8c62-08fae7e26c8c","Type":"ContainerStarted","Data":"434fe141ae25b615e8cca1be5d687ddf2af7f855bc554239f4b26a32ae5096e9"} Oct 31 02:55:14.370693 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:14.370640 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8" event={"ID":"88c4d958-b761-4b20-b3dd-c02e87b8b10c","Type":"ContainerStarted","Data":"ab637ea3ade080ca0cf898e7442782c7fb2722ff45e1d7d5d3b6d0905833cb2a"} Oct 31 02:55:14.388735 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:14.388674 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cqgsq" podStartSLOduration=1.756209449 podStartE2EDuration="7.388656924s" totalImagesPullingTime="5.632447475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:14.387408922 +0000 UTC m=+36.824070948" watchObservedRunningTime="2026-10-31 02:55:14.388656924 +0000 UTC m=+36.825318955" Oct 31 02:55:14.454342 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:14.454304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:14.454509 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:14.454369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:14.454509 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:14.454469 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:14.454509 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:14.454505 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 02:55:14.454672 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:14.454541 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:16.454525487 +0000 UTC m=+38.891187493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:14.454672 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:14.454556 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:16.454549855 +0000 UTC m=+38.891211856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : configmap "kube-rbac-proxy" not found Oct 31 02:55:15.765225 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:15.765186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:15.765665 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:15.765325 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:15.765665 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:15.765419 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls podName:810e723f-fc3a-4477-a30d-dc086bd338b1 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:23.765398111 +0000 UTC m=+46.202060115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls") pod "dns-default-hwt9g" (UID: "810e723f-fc3a-4477-a30d-dc086bd338b1") : secret "dns-default-metrics-tls" not found Oct 31 02:55:16.380324 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:16.380283 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8" event={"ID":"88c4d958-b761-4b20-b3dd-c02e87b8b10c","Type":"ContainerStarted","Data":"82f044f431c117e4e25fbde05e9b4dcc90f3e846ba6c15e95f14230ca14ee313"} Oct 31 02:55:16.382208 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:16.382178 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kjmq9" event={"ID":"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0","Type":"ContainerStarted","Data":"f15d7bbbfcc593e21b548d68ff7cce809a54b81b62a577290fd6028b2ede40ae"} Oct 31 02:55:16.382366 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:16.382322 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:16.416303 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:16.416228 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8" podStartSLOduration=1.720302818 podStartE2EDuration="4.416210879s" totalImagesPullingTime="2.695908061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:16.398706249 +0000 UTC m=+38.835368274" watchObservedRunningTime="2026-10-31 02:55:16.416210879 +0000 UTC m=+38.852872905" Oct 31 02:55:16.416677 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:16.416635 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kjmq9" podStartSLOduration=34.408324925 podStartE2EDuration="38.416621629s" totalImagesPullingTime="4.008296704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:16.416149945 +0000 UTC m=+38.852811967" watchObservedRunningTime="2026-10-31 02:55:16.416621629 +0000 UTC m=+38.853283656" Oct 31 02:55:16.471495 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:16.471413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:16.471495 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:16.471468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:16.471699 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:16.471585 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:16.471699 ip-10-0-129-107 kubenswrapper[2577]: E1031 02:55:16.471668 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:20.471646804 +0000 UTC m=+42.908308811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:16.472072 ip-10-0-129-107 kubenswrapper[2577]: I1031 02:55:16.472052 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:18.243243 ip-10-0-129-107 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 02:55:18.255358 ip-10-0-129-107 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 02:55:18.255577 ip-10-0-129-107 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 02:55:18.255636 ip-10-0-129-107 systemd[1]: kubelet.service: Consumed 2.232s CPU time. Oct 31 02:55:18.273702 ip-10-0-129-107 systemd[1]: Starting Kubernetes Kubelet... Oct 31 02:55:18.356497 ip-10-0-129-107 kubenswrapper[4456]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:55:18.356497 ip-10-0-129-107 kubenswrapper[4456]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 02:55:18.356497 ip-10-0-129-107 kubenswrapper[4456]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:55:18.356497 ip-10-0-129-107 kubenswrapper[4456]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:55:18.358593 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358508 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:18.358593 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358593 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358597 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358600 4456 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358604 4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358607 4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358610 4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358613 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358616 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358618 4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358621 4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358623 4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358626 4456 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358629 4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358632 4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358634 4456 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358638 4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358640 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358643 4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358645 4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358648 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:18.358663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358651 4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358654 4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358657 4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358659 4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358662 4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358665 4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358668 4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358670 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358674 4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358677 4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358680 4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358682 4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358685 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358687 4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358690 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358693 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358695 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358698 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358701 4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:18.359156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358704 4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358706 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358709 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358711 4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358714 4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358717 4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358719 4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358722 4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358724 4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358727 4456 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358730 4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358733 4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358736 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358739 4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358741 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358744 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358749 4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358752 4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358755 4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:18.359616 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358758 4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358761 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358764 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358766 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358769 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358772 4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358781 4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358784 4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358787 4456 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358789 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358792 4456 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358800 4456 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358803 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358806 4456 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358809 4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358812 4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358814 4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358817 4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358820 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358823 4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:18.360096 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358826 4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358828 4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358831 4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358834 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358837 4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358840 4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358843 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358845 4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358848 4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358850 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358853 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358855 4456 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358858 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358860 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358863 4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358866 4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358868 4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358871 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358873 4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:18.360582 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358878 4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358883 4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358886 4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358889 4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358892 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358898 4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358901 4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.358904 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359714 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359722 4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359725 4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359728 4456 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359732 4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359734 4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359737 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359740 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359742 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359745 4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359748 4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:18.361044 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359751 4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359753 4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359756 4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359758 4456 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359761 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359764 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359772 4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359775 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359778 4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359780 4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359783 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359786 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359798 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359801 4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359811 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359814 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359818 4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359820 4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359830 4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359833 4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:18.361505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359836 4456 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359838 4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359841 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359844 4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359847 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359850 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359852 4456 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359855 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359857 4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359860 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359863 4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359865 4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359868 4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359871 4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359873 4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359877 4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359880 4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359883 4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359885 4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359888 4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:18.361990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359891 4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359893 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359896 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359899 4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359902 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359904 4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359907 4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359915 4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359918 4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359920 4456 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359923 4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359926 4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359929 4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359931 4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359934 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359937 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359940 4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359942 4456 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359945 4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:18.362527 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359949 4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359953 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359957 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359959 4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359962 4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359965 4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359968 4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359970 4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359973 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359975 4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359978 4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359981 4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359983 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359986 4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359988 4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359991 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359994 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.359997 4456 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360000 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360002 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:18.362990 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360005 4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360013 4456 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360016 4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360019 4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360022 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360025 4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360028 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360031 4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360046 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360049 4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360051 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360054 4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360057 4456 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360065 4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360069 4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360073 4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360558 4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360565 4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360568 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:18.363478 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360571 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360574 4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360576 4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360579 4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360582 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360585 4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360588 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360591 4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360594 4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360596 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360601 4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360604 4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360607 4456 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360610 4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360613 4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360616 4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360619 4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360621 4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360624 4456 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:18.363949 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360627 4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360630 4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360632 4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360635 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360638 4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360641 4456 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360644 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360646 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360649 4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360652 4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360654 4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360657 4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360660 4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360662 4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360665 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360667 4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360670 4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360677 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360679 4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360682 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:18.364436 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360685 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360687 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360690 4456 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360693 4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360696 4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360699 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360702 4456 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360704 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360707 4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360710 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360713 4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360716 4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360719 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360722 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360724 4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360728 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360731 4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360734 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360737 4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360739 4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:18.365049 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360742 4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360745 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360747 4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360750 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360752 4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360755 4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360758 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360761 4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360763 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360767 4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360770 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360773 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360775 4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360778 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360780 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360783 4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360786 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360788 4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360791 4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:18.365828 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360793 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360796 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360809 4456 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360812 4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360814 4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360817 4456 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360820 4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360823 4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360826 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360828 4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360831 4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360833 4456 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360836 4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360839 4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360841 4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360844 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360847 4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360849 4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360851 4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360854 4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:18.366350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360857 4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360860 4456 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360864 4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360866 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.360870 4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.360973 4456 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.360981 4456 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.360988 4456 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.360992 4456 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.360997 4456 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361000 4456 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361004 4456 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361009 4456 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361012 4456 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361015 4456 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361020 4456 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361023 4456 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361027 4456 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361030 4456 flags.go:64] FLAG: --cgroup-root="" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361044 4456 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361048 4456 flags.go:64] FLAG: --client-ca-file="" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361051 4456 flags.go:64] FLAG: --cloud-provider="external" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361055 4456 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361059 4456 flags.go:64] FLAG: --cluster-domain="" Oct 31 02:55:18.366833 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361062 4456 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361066 4456 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361069 4456 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361072 4456 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361076 4456 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361079 4456 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361082 4456 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361086 4456 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361089 4456 flags.go:64] FLAG: --contention-profiling="false" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361092 4456 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361097 4456 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361100 4456 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361103 4456 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361109 4456 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361112 4456 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361115 4456 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361118 4456 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361121 4456 flags.go:64] FLAG: --enable-server="true" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361125 4456 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361130 4456 flags.go:64] FLAG: --event-burst="100" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361133 4456 flags.go:64] FLAG: --event-qps="50" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361136 4456 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361139 4456 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361143 4456 flags.go:64] FLAG: --eviction-hard="" Oct 31 02:55:18.367417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361148 4456 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361151 4456 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361154 4456 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361157 4456 flags.go:64] FLAG: --eviction-soft="" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361161 4456 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361163 4456 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361166 4456 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361169 4456 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361172 4456 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361175 4456 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361178 4456 flags.go:64] FLAG: --feature-gates="" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361183 4456 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361186 4456 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361189 4456 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361193 4456 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361196 4456 flags.go:64] FLAG: --healthz-port="10248" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361199 4456 flags.go:64] FLAG: --help="false" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361202 4456 flags.go:64] FLAG: --hostname-override="ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361207 4456 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361210 4456 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361214 4456 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361219 4456 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361222 4456 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 02:55:18.368053 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361225 4456 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361228 4456 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361231 4456 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361234 4456 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361237 4456 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361241 4456 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361244 4456 flags.go:64] FLAG: --kube-reserved="" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361247 4456 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361250 4456 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361253 4456 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361256 4456 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361259 4456 flags.go:64] FLAG: --lock-file="" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361262 4456 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361266 4456 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361269 4456 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361274 4456 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361277 4456 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361280 4456 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361283 4456 flags.go:64] FLAG: --logging-format="text" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361286 4456 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361290 4456 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361293 4456 flags.go:64] FLAG: --manifest-url="" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361296 4456 flags.go:64] FLAG: --manifest-url-header="" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361300 4456 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361303 4456 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 02:55:18.368597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361307 4456 flags.go:64] FLAG: --max-pods="110" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361310 4456 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361315 4456 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361318 4456 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361321 4456 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361325 4456 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361328 4456 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361331 4456 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361338 4456 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361342 4456 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361345 4456 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361348 4456 flags.go:64] FLAG: --pod-cidr="" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361351 4456 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361354 4456 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361357 4456 flags.go:64] FLAG: --pods-per-core="0" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361360 4456 flags.go:64] FLAG: --port="10250" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361364 4456 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361367 4456 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-04b499806d0e3d430" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361370 4456 flags.go:64] FLAG: --qos-reserved="" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361374 4456 flags.go:64] FLAG: --read-only-port="10255" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361377 4456 flags.go:64] FLAG: --register-node="true" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361380 4456 flags.go:64] FLAG: --register-with-taints="" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361383 4456 flags.go:64] FLAG: --registry-burst="10" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361386 4456 flags.go:64] FLAG: --registry-qps="5" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361389 4456 flags.go:64] FLAG: --reserved-cpus="" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361392 4456 flags.go:64] FLAG: --reserved-memory="" Oct 31 02:55:18.369211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361396 4456 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361399 4456 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361402 4456 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361405 4456 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361408 4456 flags.go:64] FLAG: --runonce="false" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361411 4456 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361415 4456 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361418 4456 flags.go:64] FLAG: --seccomp-default="false" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361422 4456 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361425 4456 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361428 4456 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361433 4456 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361436 4456 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361439 4456 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361442 4456 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361445 4456 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361448 4456 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361451 4456 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361454 4456 flags.go:64] FLAG: --system-cgroups="" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361457 4456 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361462 4456 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361465 4456 flags.go:64] FLAG: --tls-cert-file="" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361468 4456 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361472 4456 flags.go:64] FLAG: --tls-min-version="" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361475 4456 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 02:55:18.369807 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361480 4456 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361483 4456 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361486 4456 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361489 4456 flags.go:64] FLAG: --v="2" Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361493 4456 flags.go:64] FLAG: --version="false" Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361497 4456 flags.go:64] FLAG: --vmodule="" Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361501 4456 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361505 4456 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361645 4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361648 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361651 4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361655 4456 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361658 4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361660 4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361663 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361668 4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361670 4456 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361673 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361676 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361679 4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361682 4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:18.370546 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361685 4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361687 4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361690 4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361692 4456 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361695 4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361698 4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361700 4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361703 4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361705 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361708 4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361711 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361713 4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361716 4456 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361719 4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361722 4456 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361725 4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361727 4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361730 4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361732 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361735 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:18.371069 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361738 4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361740 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361743 4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361746 4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361748 4456 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361751 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361755 4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361758 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361761 4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361763 4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361766 4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361769 4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361772 4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361774 4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361777 4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361779 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361782 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361785 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361788 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:18.371548 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361790 4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361793 4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361796 4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361798 4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361801 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361804 4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361807 4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361810 4456 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361812 4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361815 4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361818 4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361820 4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361823 4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361825 4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361828 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361831 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361834 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361836 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361840 4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:18.372045 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361846 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361849 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361852 4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361855 4456 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361858 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361862 4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361865 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361867 4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361870 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361873 4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361876 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361879 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361881 4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361884 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361886 4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361889 4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361892 4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361897 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361900 4456 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361903 4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:18.372768 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361906 4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361909 4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361911 4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361914 4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361916 4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361920 4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361923 4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361925 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361929 4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361933 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361937 4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361939 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361944 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361946 4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.361949 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:18.373366 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.361975 4456 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362011 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362016 4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362019 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362022 4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362025 4456 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362029 4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362031 4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362046 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362049 4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362052 4456 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362055 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362058 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362060 4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362065 4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362067 4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362070 4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362073 4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362076 4456 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:18.373829 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362079 4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362081 4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362084 4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362087 4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362089 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362092 4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362095 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362097 4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362100 4456 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362103 4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362105 4456 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362109 4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362112 4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362115 4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362117 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362120 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362123 4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362125 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362128 4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362131 4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:18.374350 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362133 4456 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362136 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362139 4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362141 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362144 4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362146 4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362149 4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362153 4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362156 4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362158 4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362161 4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362164 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362167 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362170 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362172 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362175 4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362177 4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362180 4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362183 4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:18.374919 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362185 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362188 4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362191 4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362193 4456 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362197 4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362200 4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362203 4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362205 4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362208 4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362210 4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362214 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362216 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362220 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362223 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362227 4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362230 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362233 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362235 4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362238 4456 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362241 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:18.375567 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362244 4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362247 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362250 4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362252 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362255 4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362257 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362261 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362263 4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362266 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362268 4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362271 4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362274 4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362277 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362279 4456 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362282 4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362284 4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362288 4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362291 4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362294 4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362296 4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:18.376170 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362299 4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362301 4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362304 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362308 4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362312 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362314 4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362318 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362321 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.362323 4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.367390 4456 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.367423 4456 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.367448 4456 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.367454 4456 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367517 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367523 4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367526 4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367530 4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367532 4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367535 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:18.376663 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367538 4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367541 4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367544 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367547 4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367550 4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367553 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367555 4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367559 4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367562 4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367565 4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367567 4456 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367570 4456 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367573 4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367575 4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367578 4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367581 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367584 4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367587 4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367590 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367593 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:18.377124 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367595 4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367598 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367602 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367604 4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367607 4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367609 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367612 4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367615 4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367617 4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367620 4456 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367622 4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367625 4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367628 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367631 4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367633 4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367636 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367639 4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367642 4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367646 4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:18.377607 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367650 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367654 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367657 4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367660 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367663 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367666 4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367669 4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367671 4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367674 4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367677 4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367679 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367682 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367685 4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367687 4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367690 4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367693 4456 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367695 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367698 4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367700 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:18.378335 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367703 4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367706 4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367708 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367711 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367714 4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367716 4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367719 4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367722 4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367724 4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367727 4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367730 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367732 4456 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367735 4456 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367737 4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367740 4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367743 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367747 4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367751 4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367754 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:18.379101 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367757 4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367760 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367763 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367766 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367769 4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367772 4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367774 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367777 4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367779 4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367782 4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367785 4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367797 4456 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367800 4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367803 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367806 4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367809 4456 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367811 4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367814 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367817 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367820 4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:18.379815 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367822 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367825 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.367828 4456 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.367856 4456 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368004 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368008 4456 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368011 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368014 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368017 4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368020 4456 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368023 4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368025 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368028 4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368031 4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368046 4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368049 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:18.380505 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368052 4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368055 4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368057 4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368060 4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368063 4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368066 4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368069 4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368072 4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368075 4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368078 4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368081 4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368083 4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368086 4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368089 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368091 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368094 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368097 4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368099 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368102 4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:18.380887 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368104 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368107 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368109 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368112 4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368116 4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368120 4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368123 4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368127 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368129 4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368132 4456 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368135 4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368138 4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368140 4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368143 4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368145 4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368148 4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368151 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368154 4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368156 4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:18.381338 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368159 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368161 4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368164 4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368166 4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368169 4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368171 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368174 4456 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368177 4456 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368179 4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368182 4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368184 4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368186 4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368189 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368192 4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368194 4456 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368197 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368199 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368202 4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368205 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368207 4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:18.381781 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368210 4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368215 4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368219 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368222 4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368225 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368228 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368231 4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368234 4456 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368236 4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368239 4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368241 4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368244 4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368247 4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368249 4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368252 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368255 4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368257 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368260 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368262 4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:18.382282 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368265 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368268 4456 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368270 4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368273 4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368275 4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368278 4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368280 4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368283 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368285 4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368288 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368291 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368294 4456 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368296 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368299 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368302 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368305 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:18.382742 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368308 4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.368331 4456 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368364 4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368368 4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368371 4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368374 4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368377 4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368380 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368382 4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368385 4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368388 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368390 4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368393 4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368396 4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368398 4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368401 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:18.383142 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368403 4456 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368406 4456 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368409 4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368411 4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368416 4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368419 4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368422 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368425 4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368428 4456 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368431 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368433 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368436 4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368438 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368441 4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368444 4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368448 4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368452 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368455 4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368458 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:18.383521 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368461 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368464 4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368467 4456 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368469 4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368472 4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368474 4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368477 4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368480 4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368482 4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368485 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368488 4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368490 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368493 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368495 4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368498 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368501 4456 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368504 4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368508 4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368512 4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368514 4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:18.383982 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368517 4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368520 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368522 4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368525 4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368528 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368531 4456 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368533 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368536 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368539 4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368542 4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368545 4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368547 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368550 4456 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368553 4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368555 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368558 4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368560 4456 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368563 4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368566 4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368568 4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:18.384510 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368570 4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368573 4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368576 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368578 4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368581 4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368583 4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368586 4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368589 4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368592 4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368596 4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368600 4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368603 4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368606 4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368609 4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368612 4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368614 4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368617 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368620 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368622 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:18.385017 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368625 4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368628 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368630 4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368633 4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368636 4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368638 4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368641 4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368645 4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368649 4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368652 4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368655 4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368658 4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368661 4456 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:18.368663 4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.368737 4456 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.369709 4456 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.369765 4456 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.369936 4456 server.go:1014] "Starting client certificate rotation" Oct 31 02:55:18.385559 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.370070 4456 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 02:55:18.386059 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.370128 4456 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 02:49:38 +0000 UTC" deadline="2028-08-07 16:14:08.951681924 +0000 UTC" Oct 31 02:55:18.386059 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.370187 4456 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15517h18m50.581497068s" Oct 31 02:55:18.386059 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.370397 4456 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:55:18.386059 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.371441 4456 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:55:18.386059 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.373232 4456 log.go:25] "Validated CRI v1 runtime API" Oct 31 02:55:18.386059 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.374877 4456 log.go:25] "Validated CRI v1 image API" Oct 31 02:55:18.386059 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.375629 4456 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 02:55:18.386059 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.377257 4456 fs.go:135] Filesystem UUIDs: map[049aa447-5ff1-4a9a-a3a6-cb611b7c2f58:/dev/nvme0n1p4 0f0aad77-171b-4b4a-a9b8-d2c86d4ebc2d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 02:55:18.386276 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.377276 4456 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08000406ce92b78184d10f9abae51dc78c7143d0e3474535035b4776e5389ee7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08000406ce92b78184d10f9abae51dc78c7143d0e3474535035b4776e5389ee7/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e6ed5c3012446197ef34ac0965c1fd24d0f507a35f16465365634c1c2b3a9ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e6ed5c3012446197ef34ac0965c1fd24d0f507a35f16465365634c1c2b3a9ec/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1078d9d7bd59fac10155569561010a84b87815f351269d5b1bdb5778c60f17ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1078d9d7bd59fac10155569561010a84b87815f351269d5b1bdb5778c60f17ed/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e4788a9e9512eb0a3edd7845396e8b3882e48d761abb29316c78bd9543d9527/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e4788a9e9512eb0a3edd7845396e8b3882e48d761abb29316c78bd9543d9527/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e16567e212d6b5b6fc2aaff2bbfed309f2661fa76034f5faa4436a6647119da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e16567e212d6b5b6fc2aaff2bbfed309f2661fa76034f5faa4436a6647119da/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b871bf82f5b0a5e62f3c2314e1584afcd81fb0526b6002912d5395d66c0721f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b871bf82f5b0a5e62f3c2314e1584afcd81fb0526b6002912d5395d66c0721f/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e7da63f88a8d647f954c00207e126de0c9d7078bd811b7513e53c95899263c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e7da63f88a8d647f954c00207e126de0c9d7078bd811b7513e53c95899263c5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84d3f2282ed849c17fdcb74923acf981d206fecde9523444386bcfcbd893502e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84d3f2282ed849c17fdcb74923acf981d206fecde9523444386bcfcbd893502e/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab637ea3ade080ca0cf898e7442782c7fb2722ff45e1d7d5d3b6d0905833cb2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab637ea3ade080ca0cf898e7442782c7fb2722ff45e1d7d5d3b6d0905833cb2a/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b88322dba7979796d4eaacb871386a58c921cf9ac8b0dfe95f7f6180c7dd09aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b88322dba7979796d4eaacb871386a58c921cf9ac8b0dfe95f7f6180c7dd09aa/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e274af9265f8b19a46910c59ab651f6be4819bd7b5cd43f95f590fb16d9802a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e274af9265f8b19a46910c59ab651f6be4819bd7b5cd43f95f590fb16d9802a8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3fdb9bb2fdcd79ea89101783c588df63c453e920ffdd19b55380b9442e12616/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3fdb9bb2fdcd79ea89101783c588df63c453e920ffdd19b55380b9442e12616/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f19e9732a1bd749e2676e0d8b714829c06324110d140810616c414f393247717/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f19e9732a1bd749e2676e0d8b714829c06324110d140810616c414f393247717/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcd97591b2f336f4047f85efd77085896f09401c9144412995da9b3073c7a659/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcd97591b2f336f4047f85efd77085896f09401c9144412995da9b3073c7a659/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0facc8a8-af14-4207-bb9f-2e36129b0343/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0facc8a8-af14-4207-bb9f-2e36129b0343/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0facc8a8-af14-4207-bb9f-2e36129b0343/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0facc8a8-af14-4207-bb9f-2e36129b0343/volumes/kubernetes.io~empty-dir/tmp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0facc8a8-af14-4207-bb9f-2e36129b0343/volumes/kubernetes.io~projected/kube-api-access-58487:{mountpoint:/var/lib/kubelet/pods/0facc8a8-af14-4207-bb9f-2e36129b0343/volumes/kubernetes.io~projected/kube-api-access-58487 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24901959-cd66-4412-a070-aeb1cfaa8444/volumes/kubernetes.io~projected/kube-api-access-g9pnv:{mountpoint:/var/lib/kubelet/pods/24901959-cd66-4412-a070-aeb1cfaa8444/volumes/kubernetes.io~projected/kube-api-access-g9pnv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24d5a9bc-29a3-42b5-bf90-9468b578b0f7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/24d5a9bc-29a3-42b5-bf90-9468b578b0f7/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2be3a160-6ada-4e51-a6bf-19745a8670f4/volumes/kubernetes.io~projected/kube-api-access-jxm7m:{mountpoint:/var/lib/kubelet/pods/2be3a160-6ada-4e51-a6bf-19745a8670f4/volumes/kubernetes.io~projected/kube-api-access-jxm7m major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b7047af-e1c4-4e1c-81ea-f777c15b0d97/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3b7047af-e1c4-4e1c-81ea-f777c15b0d97/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4249fb31-38de-4684-abd0-6d49c6dacd59/volumes/kubernetes.io~projected/kube-api-access-249fn:{mountpoint:/var/lib/kubelet/pods/4249fb31-38de-4684-abd0-6d49c6dacd59/volumes/kubernetes.io~projected/kube-api-access-249fn major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ded5081-b545-47f2-be60-975cf616f05f/volumes/kubernetes.io~projected/kube-api-access-gdh6m:{mountpoint:/var/lib/kubelet/pods/4ded5081-b545-47f2-be60-975cf616f05f/volumes/kubernetes.io~projected/kube-api-access-gdh6m major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fd7955b-02fe-411e-83a4-abac6fdbb242/volumes/kubernetes.io~projected/kube-api-access-zdx4z:{mountpoint:/var/lib/kubelet/pods/6fd7955b-02fe-411e-83a4-abac6fdbb242/volumes/kubernetes.io~projected/kube-api-access-zdx4z major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79426555-723a-431c-862c-8f936334be71/volumes/kubernetes.io~projected/kube-api-access-4l65r:{mountpoint:/var/lib/kubelet/pods/79426555-723a-431c-862c-8f936334be71/volumes/kubernetes.io~projected/kube-api-access-4l65r major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/810e723f-fc3a-4477-a30d-dc086bd338b1/volumes/kubernetes.io~projected/kube-api-access-jvb7x:{mountpoint:/var/lib/kubelet/pods/810e723f-fc3a-4477-a30d-dc086bd338b1/volumes/kubernetes.io~projected/kube-api-access-jvb7x major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88c4d958-b761-4b20-b3dd-c02e87b8b10c/volumes/kubernetes.io~projected/kube-api-access-t4wsb:{mountpoint:/var/lib/kubelet/pods/88c4d958-b761-4b20-b3dd-c02e87b8b10c/volumes/kubernetes.io~projected/kube-api-access-t4wsb major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af399405-2475-49b9-9fdd-7f43dff044b6/volumes/kubernetes.io~projected/kube-api-access-sskj6:{mountpoint:/var/lib/kubelet/pods/af399405-2475-49b9-9fdd-7f43dff044b6/volumes/kubernetes.io~projected/kube-api-access-sskj6 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4bb32ff-188e-4471-8c62-08fae7e26c8c/volumes/kubernetes.io~projected/kube-api-access-mv4ll:{mountpoint:/var/lib/kubelet/pods/b4bb32ff-188e-4471-8c62-08fae7e26c8c/volumes/kubernetes.io~projected/kube-api-access-mv4ll major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4bb32ff-188e-4471-8c62-08fae7e26c8c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b4bb32ff-188e-4471-8c62-08fae7e26c8c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c05a3117-b24e-4810-9a49-2dd3cc9cc88b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c05a3117-b24e-4810-9a49-2dd3cc9cc88b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c05a3117-b24e-4810-9a49-2dd3cc9cc88b/volumes/kubernetes.io~projected/kube-api-access-lx4vt:{mountpoint:/var/lib/kubelet/pods/c05a3117-b24e-4810-9a49-2dd3cc9cc88b/volumes/kubernetes.io~projected/kube-api-access-lx4vt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c05a3117-b24e-4810-9a49-2dd3cc9cc88b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c05a3117-b24e-4810-9a49-2dd3cc9cc88b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0/volumes/kubernetes.io~projected/kube-api-access-vnm92:{mountpoint:/var/lib/kubelet/pods/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0/volumes/kubernetes.io~projected/kube-api-access-vnm92 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee641d86-9b04-4f02-a22b-7cc55694a018/volumes/kubernetes.io~projected/kube-api-access-9jwx7:{mountpoint:/var/lib/kubelet/pods/ee641d86-9b04-4f02-a22b-7cc55694a018/volumes/kubernetes.io~projected/kube-api-access-9jwx7 major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1c2db47acd5329b6a1ec43d18056f023f1667705dc0837a69ab1bdc205c4a586/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/68f51ec2543c8d83e1d7d3da3abf884f149e51b2c5cfaa5141d3e989e913040f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e0cb659dde19be40b169e79b864906ad23423da7662bd182368618d6d3be232e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/3507266476e18f29d6de7c446943c50ecbc17ebcde280c1d779d24c8f1538ba4/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4f5a5708e257c0919e4fe952fdea965409e0652a7ab5f59158a56d276b692cf7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2787bce38df303093c487e8d0d2e33029850923daac4a9ca35c1ad9b53246271/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ecd83b1b0cb778e7690d2ce3526699cddb48f431912f17e3842a875033ee6bac/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/48360f39362e3bc99c82767b9dcaa6a7ab9b69bb742bf6419e2ad6a287b31b86/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/81635012798a8bd63da52d52192c33d9d13679eb6ba48e5eb1f616b0517a82e1/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f26b0a6602bedb2a618268eaf2bb15955fb7b2987befbbf5be40c38c83ff41cc/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a46af6e8760aad5b4d56d98db936e7b46731c19686e63109537b26799b313ed4/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/7e95f5de62ed6b8fa3aca12a321c15a232fd6e1c481aab04aadc104a5c28c452/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d443aa4d233797bd7378d7f51e09ced9241f24c2601804abb16d03977b2b8a37/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0c8cf93f82976b13bc4c6c179be12a287114b77f1d2d4233767c605c0db30aa3/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/12954b61e8df6998b35ca365124f4ab7f2347d62bc6ed06fcef14e45987e3524/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cb19f8ac992eeb481189ac63500d389870c5598715c96e6d2ca8b7ef5b8bd9d7/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/02101a73dfc472c4487c5ca645292c543456e544d90c3efe6a4145dc2f52223b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9115c99dc2ae8e051be9016b0e8fa3a3d1585627dbe4b6404249175aadbc658e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/467cbf62037038466560c3e78a0fb20ed5ac512b1c539626512b295527611383/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f329cfe11a110834da29281155e470aec43bd5ae0ff3059ab11f0f9474a1e11b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/61c2e2e62be566ccf408060560cc2f65ca0e327149c1e147c234663e9feae211/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/2c06b3416bb49252854ec769900160ee9725575b4559902197d9e72dd44ef0ec/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/229792dd9d1fd55f3f92aec7e7645989a1e416d682c420aa957e9ae848d7b7db/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/afea6119ec110edb9964cac4db8c16a77bff0c21604c19babce7e6ed3afc91a4/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/0d9ee00d9cd75d6b8499b4789cc4aa8886e19c18da021c4f434dce34a4a762f3/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/663290ba2c8acf9989717ad5d23985ad3e2658233f534a160cf3a7cebb6432cd/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/399b79cc8124643d72fdf179471bc7b33597b07d52bd95579ea31e4e5c6b37c3/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/17a2da2df8cdc6584a78e05aa159a2bd666111e01116c89bcb174db67746e2c4/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/0f805e15e03034faca6929af61d5d2ddd4e7a33a96ab51c1425170cc37674ec2/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/e1e9ad5ddde6d29cf1508f5413140385b10945ffe898dfd0c7b85e9d223be8f2/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/1cdd1a502e1bec11ebc331980208538822acef4df8c28b3e0656a41e79cf54b1/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/afb63d7ee0efbc9d2dce58e977b4e37e5a58119cd4660c725bcb5adc0707aba5/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e7e979fd43d50e18b918fa98e1f3392eb2af4d8ed89a3bce859abe5c7a4a6b13/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/33fb4adcf0f9f50de17a7a39f4dcef891b8d0fa803c5b0f9ac890d3a8a40ec0c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/208eef998f58c5cc63b5da39f7146e513637dfc784992bfb583d44e7c7f5a121/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/499348c2311fc7c24b6e8e7abd53a93caad28e9d95f1398b3d2e2e29bf570aa8/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f07a5f9990e74b45e7587e86f59b646a22c4fae0a5aaca70126452621be6d929/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/f4d4bbe6c6e05b1ea62c83fa25c9687f8940855b39c4f353b07510063588c68d/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ef3e4c4b27fe61b4d39d6f39bd09787e5442d6f54302a5b533dda6edc7dfebe1/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.384093 4456 manager.go:217] Machine: {Timestamp:2026-10-31 02:55:18.383596207 +0000 UTC m=+0.064122325 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099954 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec238c92b3df410465943d5f9dfa6d2d SystemUUID:ec238c92-b3df-4104-6594-3d5f9dfa6d2d BootID:933e9d3d-8dae-4436-9596-8a6f7be3b816 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1078d9d7bd59fac10155569561010a84b87815f351269d5b1bdb5778c60f17ed/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c05a3117-b24e-4810-9a49-2dd3cc9cc88b/volumes/kubernetes.io~projected/kube-api-access-lx4vt DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08000406ce92b78184d10f9abae51dc78c7143d0e3474535035b4776e5389ee7/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e4788a9e9512eb0a3edd7845396e8b3882e48d761abb29316c78bd9543d9527/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c05a3117-b24e-4810-9a49-2dd3cc9cc88b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2be3a160-6ada-4e51-a6bf-19745a8670f4/volumes/kubernetes.io~projected/kube-api-access-jxm7m DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24901959-cd66-4412-a070-aeb1cfaa8444/volumes/kubernetes.io~projected/kube-api-access-g9pnv DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b88322dba7979796d4eaacb871386a58c921cf9ac8b0dfe95f7f6180c7dd09aa/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24d5a9bc-29a3-42b5-bf90-9468b578b0f7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0facc8a8-af14-4207-bb9f-2e36129b0343/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0facc8a8-af14-4207-bb9f-2e36129b0343/volumes/kubernetes.io~projected/kube-api-access-58487 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b7047af-e1c4-4e1c-81ea-f777c15b0d97/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e274af9265f8b19a46910c59ab651f6be4819bd7b5cd43f95f590fb16d9802a8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c05a3117-b24e-4810-9a49-2dd3cc9cc88b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84d3f2282ed849c17fdcb74923acf981d206fecde9523444386bcfcbd893502e/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f19e9732a1bd749e2676e0d8b714829c06324110d140810616c414f393247717/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/88c4d958-b761-4b20-b3dd-c02e87b8b10c/volumes/kubernetes.io~projected/kube-api-access-t4wsb DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b871bf82f5b0a5e62f3c2314e1584afcd81fb0526b6002912d5395d66c0721f/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ded5081-b545-47f2-be60-975cf616f05f/volumes/kubernetes.io~projected/kube-api-access-gdh6m DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6fd7955b-02fe-411e-83a4-abac6fdbb242/volumes/kubernetes.io~projected/kube-api-access-zdx4z DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4249fb31-38de-4684-abd0-6d49c6dacd59/volumes/kubernetes.io~projected/kube-api-access-249fn DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e6ed5c3012446197ef34ac0965c1fd24d0f507a35f16465365634c1c2b3a9ec/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0/volumes/kubernetes.io~projected/kube-api-access-vnm92 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/0facc8a8-af14-4207-bb9f-2e36129b0343/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3fdb9bb2fdcd79ea89101783c588df63c453e920ffdd19b55380b9442e12616/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4bb32ff-188e-4471-8c62-08fae7e26c8c/volumes/kubernetes.io~projected/kube-api-access-mv4ll DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af399405-2475-49b9-9fdd-7f43dff044b6/volumes/kubernetes.io~projected/kube-api-access-sskj6 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcd97591b2f336f4047f85efd77085896f09401c9144412995da9b3073c7a659/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/810e723f-fc3a-4477-a30d-dc086bd338b1/volumes/kubernetes.io~projected/kube-api-access-jvb7x DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79426555-723a-431c-862c-8f936334be71/volumes/kubernetes.io~projected/kube-api-access-4l65r DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e7da63f88a8d647f954c00207e126de0c9d7078bd811b7513e53c95899263c5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4bb32ff-188e-4471-8c62-08fae7e26c8c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee641d86-9b04-4f02-a22b-7cc55694a018/volumes/kubernetes.io~projected/kube-api-access-9jwx7 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e16567e212d6b5b6fc2aaff2bbfed309f2661fa76034f5faa4436a6647119da/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab637ea3ade080ca0cf898e7442782c7fb2722ff45e1d7d5d3b6d0905833cb2a/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:84d3f2282ed849c MacAddress:3e:c6:3a:2a:a5:42 Speed:10000 Mtu:8901} {Name:ab637ea3ade080c MacAddress:7a:a0:bc:a3:55:05 Speed:10000 Mtu:8901} {Name:b88322dba797979 MacAddress:82:ca:51:c8:ac:a6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f6:82:5a:2f Speed:0 Mtu:9001} {Name:br-int MacAddress:e2:2f:98:57:92:dc Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f6:82:5a:2f Speed:0 Mtu:9001} {Name:f19e9732a1bd749 MacAddress:96:cf:03:f3:82:0f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:da:30:3d:ae:70:61 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0e:c8:06:12:c0:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.384339 4456 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.384383 4456 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.384732 4456 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.384750 4456 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-107.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.384881 4456 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.384889 4456 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.384899 4456 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.384914 4456 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.384984 4456 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.385147 4456 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.385251 4456 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.385284 4456 kubelet.go:493] "Attempting to sync node with API server" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.385294 4456 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.385303 4456 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.385312 4456 kubelet.go:405] "Adding apiserver pod source" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.385324 4456 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.386108 4456 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 02:55:18.386502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.386123 4456 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:55:18.387196 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.386588 4456 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 02:55:18.388479 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388465 4456 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 02:55:18.388479 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388478 4456 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:55:18.388587 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388577 4456 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 02:55:18.388763 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388753 4456 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 02:55:18.388798 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388781 4456 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 02:55:18.388881 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388872 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 02:55:18.388909 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388886 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 02:55:18.388909 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388894 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 02:55:18.388909 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388900 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 02:55:18.388909 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388905 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 02:55:18.389012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388911 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 02:55:18.389012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388917 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 02:55:18.389012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388922 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 02:55:18.389012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388929 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 02:55:18.389012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388934 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 02:55:18.389012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388952 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 02:55:18.389012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388962 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 02:55:18.389012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388985 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 02:55:18.389012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.388991 4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 02:55:18.392400 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.392385 4456 server.go:1263] "Started kubelet" Oct 31 02:55:18.392494 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.392450 4456 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 02:55:18.392958 ip-10-0-129-107 systemd[1]: Started Kubernetes Kubelet. Oct 31 02:55:18.395918 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.394214 4456 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 02:55:18.396022 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.395921 4456 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 02:55:18.396670 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.396609 4456 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 02:55:18.396873 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.396855 4456 server.go:317] "Adding debug handlers to kubelet server" Oct 31 02:55:18.398207 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.398134 4456 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 02:55:18.398286 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.398213 4456 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 02:55:18.398348 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.398300 4456 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 02:49:40 +0000 UTC" deadline="2028-06-10 19:51:42.13797062 +0000 UTC" Oct 31 02:55:18.398348 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.398321 4456 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14128h56m23.73965288s" Oct 31 02:55:18.398348 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:18.398335 4456 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-107.us-east-2.compute.internal\" not found" Oct 31 02:55:18.399722 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.399696 4456 factory.go:55] Registering systemd factory Oct 31 02:55:18.399722 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.399715 4456 factory.go:223] Registration of the systemd container factory successfully Oct 31 02:55:18.401184 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.399935 4456 factory.go:153] Registering CRI-O factory Oct 31 02:55:18.401184 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.399951 4456 factory.go:223] Registration of the crio container factory successfully Oct 31 02:55:18.401184 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.400006 4456 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 02:55:18.401184 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.400025 4456 factory.go:103] Registering Raw factory Oct 31 02:55:18.401184 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.400057 4456 manager.go:1199] Started watching for new ooms in manager Oct 31 02:55:18.401184 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.400077 4456 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 02:55:18.401184 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.400378 4456 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 02:55:18.401184 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.400393 4456 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 02:55:18.401184 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.400418 4456 manager.go:319] Starting recovery of all containers Oct 31 02:55:18.402095 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402020 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4249fb31-38de-4684-abd0-6d49c6dacd59" volumeName="kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 02:55:18.402095 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402080 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79426555-723a-431c-862c-8f936334be71" volumeName="kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-cni-binary-copy" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402097 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="810e723f-fc3a-4477-a30d-dc086bd338b1" volumeName="kubernetes.io/empty-dir/810e723f-fc3a-4477-a30d-dc086bd338b1-tmp-dir" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402112 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4bb32ff-188e-4471-8c62-08fae7e26c8c" volumeName="kubernetes.io/projected/b4bb32ff-188e-4471-8c62-08fae7e26c8c-kube-api-access-mv4ll" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402125 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0facc8a8-af14-4207-bb9f-2e36129b0343" volumeName="kubernetes.io/projected/0facc8a8-af14-4207-bb9f-2e36129b0343-kube-api-access-58487" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402139 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2be3a160-6ada-4e51-a6bf-19745a8670f4" volumeName="kubernetes.io/configmap/2be3a160-6ada-4e51-a6bf-19745a8670f4-serviceca" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402152 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fd7955b-02fe-411e-83a4-abac6fdbb242" volumeName="kubernetes.io/projected/6fd7955b-02fe-411e-83a4-abac6fdbb242-kube-api-access-zdx4z" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402166 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee641d86-9b04-4f02-a22b-7cc55694a018" volumeName="kubernetes.io/empty-dir/ee641d86-9b04-4f02-a22b-7cc55694a018-tmp-dir" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402183 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24d5a9bc-29a3-42b5-bf90-9468b578b0f7" volumeName="kubernetes.io/secret/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-agent-certs" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402197 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4249fb31-38de-4684-abd0-6d49c6dacd59" volumeName="kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402210 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ded5081-b545-47f2-be60-975cf616f05f" volumeName="kubernetes.io/projected/4ded5081-b545-47f2-be60-975cf616f05f-kube-api-access-gdh6m" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402223 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79426555-723a-431c-862c-8f936334be71" volumeName="kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-multus-daemon-config" seLinuxMountContext="" Oct 31 02:55:18.402231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402235 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="810e723f-fc3a-4477-a30d-dc086bd338b1" volumeName="kubernetes.io/configmap/810e723f-fc3a-4477-a30d-dc086bd338b1-config-volume" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402247 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="810e723f-fc3a-4477-a30d-dc086bd338b1" volumeName="kubernetes.io/projected/810e723f-fc3a-4477-a30d-dc086bd338b1-kube-api-access-jvb7x" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402265 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0facc8a8-af14-4207-bb9f-2e36129b0343" volumeName="kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-tmp" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402279 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24901959-cd66-4412-a070-aeb1cfaa8444" volumeName="kubernetes.io/projected/24901959-cd66-4412-a070-aeb1cfaa8444-kube-api-access-g9pnv" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402320 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4249fb31-38de-4684-abd0-6d49c6dacd59" volumeName="kubernetes.io/projected/4249fb31-38de-4684-abd0-6d49c6dacd59-kube-api-access-249fn" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402338 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ded5081-b545-47f2-be60-975cf616f05f" volumeName="kubernetes.io/empty-dir/4ded5081-b545-47f2-be60-975cf616f05f-data-volume" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402351 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c05a3117-b24e-4810-9a49-2dd3cc9cc88b" volumeName="kubernetes.io/projected/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-kube-api-access-lx4vt" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402362 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2be3a160-6ada-4e51-a6bf-19745a8670f4" volumeName="kubernetes.io/projected/2be3a160-6ada-4e51-a6bf-19745a8670f4-kube-api-access-jxm7m" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402375 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7047af-e1c4-4e1c-81ea-f777c15b0d97" volumeName="kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402386 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ded5081-b545-47f2-be60-975cf616f05f" volumeName="kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402410 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79426555-723a-431c-862c-8f936334be71" volumeName="kubernetes.io/projected/79426555-723a-431c-862c-8f936334be71-kube-api-access-4l65r" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402421 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af399405-2475-49b9-9fdd-7f43dff044b6" volumeName="kubernetes.io/projected/af399405-2475-49b9-9fdd-7f43dff044b6-kube-api-access-sskj6" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402432 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4bb32ff-188e-4471-8c62-08fae7e26c8c" volumeName="kubernetes.io/secret/b4bb32ff-188e-4471-8c62-08fae7e26c8c-cert" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402470 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c05a3117-b24e-4810-9a49-2dd3cc9cc88b" volumeName="kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-script-lib" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402483 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c05a3117-b24e-4810-9a49-2dd3cc9cc88b" volumeName="kubernetes.io/secret/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402496 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0facc8a8-af14-4207-bb9f-2e36129b0343" volumeName="kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-tuned" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402512 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24d5a9bc-29a3-42b5-bf90-9468b578b0f7" volumeName="kubernetes.io/configmap/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-konnectivity-ca" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402523 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e02f7dec-7aaf-4418-b520-74ce9c2ebbf0" volumeName="kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402534 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee641d86-9b04-4f02-a22b-7cc55694a018" volumeName="kubernetes.io/projected/ee641d86-9b04-4f02-a22b-7cc55694a018-kube-api-access-9jwx7" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402546 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fd7955b-02fe-411e-83a4-abac6fdbb242" volumeName="kubernetes.io/configmap/6fd7955b-02fe-411e-83a4-abac6fdbb242-iptables-alerter-script" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402558 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c05a3117-b24e-4810-9a49-2dd3cc9cc88b" volumeName="kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-env-overrides" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402570 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c4d958-b761-4b20-b3dd-c02e87b8b10c" volumeName="kubernetes.io/projected/88c4d958-b761-4b20-b3dd-c02e87b8b10c-kube-api-access-t4wsb" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402582 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c05a3117-b24e-4810-9a49-2dd3cc9cc88b" volumeName="kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-config" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402593 4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4249fb31-38de-4684-abd0-6d49c6dacd59" volumeName="kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-binary-copy" seLinuxMountContext="" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402604 4456 reconstruct.go:97] "Volume reconstruction finished" Oct 31 02:55:18.402711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.402611 4456 reconciler.go:29] "Reconciler: start to sync state" Oct 31 02:55:18.406720 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.406697 4456 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:18.408743 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.408688 4456 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:18.409997 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.409913 4456 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:18.412576 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.412558 4456 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 02:55:18.425870 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.425827 4456 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 02:55:18.427482 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.427455 4456 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 02:55:18.427482 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.427484 4456 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 02:55:18.427616 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.427509 4456 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 02:55:18.427616 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:18.427549 4456 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 02:55:18.432564 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.432538 4456 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:18.442953 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.442925 4456 generic.go:356] "Generic (PLEG): container finished" podID="b7f3d03d6994989fb7ae18d976e5b40a" containerID="6def1099e5940d5b3d545f521e61bd9567042d4ef0fb0a05d9a91c544958b404" exitCode=0 Oct 31 02:55:18.460802 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.460772 4456 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="64f811666cfc70e12368df1bd481031a41a97a8e5f9d4a0c9a1ef829efc2905e" exitCode=0 Oct 31 02:55:18.460802 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.460798 4456 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="d1989f01e4d618ef3c28dd8e122b1ad910b2a7eb88874a0b677df6a2bc2f3644" exitCode=0 Oct 31 02:55:18.460959 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.460807 4456 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="be4e08ec20d65fe748e1c466007bdec55bcd5f0edae7a9750d17ea72b71976e2" exitCode=0 Oct 31 02:55:18.460959 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.460814 4456 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="408219afa6fab4174ea6bf82454d3074329c76e00c2398e7fefba42306eb8653" exitCode=0 Oct 31 02:55:18.460959 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.460819 4456 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="55ee6ab30963ac6809387781778e51e53970990f0dd0270e029dae47a2ef83e5" exitCode=0 Oct 31 02:55:18.460959 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.460826 4456 generic.go:356] "Generic (PLEG): container finished" podID="4249fb31-38de-4684-abd0-6d49c6dacd59" containerID="cdcf74d2f00011601d074c2e4b8df0eb6c6d1fd13ecea8c7cbcc01c2bf9d5a3b" exitCode=0 Oct 31 02:55:18.465780 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.465765 4456 manager.go:324] Recovery completed Oct 31 02:55:18.479923 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.479900 4456 cpu_manager.go:226] "Starting" policy="none" Oct 31 02:55:18.479923 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.479915 4456 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 02:55:18.480068 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.479935 4456 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 02:55:18.480133 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.480118 4456 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 02:55:18.480193 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.480134 4456 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 02:55:18.480193 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.480158 4456 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 02:55:18.480193 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.480167 4456 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 02:55:18.480193 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.480175 4456 policy_none.go:50] "Start" Oct 31 02:55:18.480193 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.480192 4456 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 02:55:18.480416 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.480205 4456 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 02:55:18.480416 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.480381 4456 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 02:55:18.480416 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.480392 4456 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 02:55:18.480416 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.480400 4456 policy_none.go:44] "Start" Oct 31 02:55:18.484170 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.484147 4456 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 02:55:18.486590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.486576 4456 manager.go:341] "Starting Device Plugin manager" Oct 31 02:55:18.487017 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:18.486600 4456 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 02:55:18.487017 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.487018 4456 server.go:87] "Starting device plugin registration server" Oct 31 02:55:18.487278 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.487266 4456 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 02:55:18.487340 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.487277 4456 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 02:55:18.487387 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.487369 4456 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 02:55:18.487495 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.487463 4456 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 02:55:18.487549 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.487504 4456 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 02:55:18.487549 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.487511 4456 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 02:55:18.487638 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.487549 4456 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T02:55:18.48749332Z","UUID":"0d5fc844-1caf-4ddd-bce9-c2ed819ec460","Handler":null,"Name":"","Endpoint":""} Oct 31 02:55:18.488770 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.488753 4456 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 02:55:18.488854 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.488779 4456 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 02:55:18.488924 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:18.488846 4456 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 02:55:18.528132 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.528045 4456 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal"] Oct 31 02:55:18.528504 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.528477 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" event={"ID":"b7f3d03d6994989fb7ae18d976e5b40a","Type":"ContainerStarted","Data":"350d706d0dfd2eee4253f84da9d407a706f93c1c2117bb5a8c31bfc6b58bfbc1"} Oct 31 02:55:18.528547 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.528511 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" event={"ID":"b7f3d03d6994989fb7ae18d976e5b40a","Type":"ContainerDied","Data":"6def1099e5940d5b3d545f521e61bd9567042d4ef0fb0a05d9a91c544958b404"} Oct 31 02:55:18.528613 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.528600 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" event={"ID":"b7f3d03d6994989fb7ae18d976e5b40a","Type":"ContainerStarted","Data":"fcd97591b2f336f4047f85efd77085896f09401c9144412995da9b3073c7a659"} Oct 31 02:55:18.528646 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.528609 4456 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.528646 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.528623 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" event={"ID":"da9a871c29174a296dbd1f4a73e3997d","Type":"ContainerStarted","Data":"445cdd83fb475b12c227e36fdfc29b1b747f10071e40246e7221dc0494687c1b"} Oct 31 02:55:18.528646 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.528641 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" event={"ID":"da9a871c29174a296dbd1f4a73e3997d","Type":"ContainerStarted","Data":"5e4788a9e9512eb0a3edd7845396e8b3882e48d761abb29316c78bd9543d9527"} Oct 31 02:55:18.528725 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.528677 4456 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.542006 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.541988 4456 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:18.542076 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:18.542030 4456 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.542554 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.542541 4456 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:18.542588 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:18.542572 4456 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.587715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.587696 4456 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:55:18.588896 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.588881 4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:55:18.588951 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.588908 4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:55:18.588951 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.588917 4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:55:18.588951 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.588945 4456 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.602760 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.602743 4456 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.602822 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.602788 4456 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.614074 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.614056 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.614162 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.614083 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.614162 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.614099 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9a871c29174a296dbd1f4a73e3997d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"da9a871c29174a296dbd1f4a73e3997d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.715115 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.715085 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.715223 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.715089 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.715223 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.715156 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.715223 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.715176 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9a871c29174a296dbd1f4a73e3997d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"da9a871c29174a296dbd1f4a73e3997d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.715223 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.715212 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f3d03d6994989fb7ae18d976e5b40a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"b7f3d03d6994989fb7ae18d976e5b40a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:18.715349 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:18.715219 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9a871c29174a296dbd1f4a73e3997d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal\" (UID: \"da9a871c29174a296dbd1f4a73e3997d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:19.386575 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.386536 4456 apiserver.go:52] "Watching apiserver" Oct 31 02:55:19.398715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.398689 4456 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 02:55:19.399097 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.399079 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-qhcg9","kube-system/global-pull-secret-syncer-vtlr6","openshift-network-operator/iptables-alerter-tq4b7","openshift-ingress-canary/ingress-canary-cqgsq","openshift-cluster-node-tuning-operator/tuned-ctx6n","openshift-multus/network-metrics-daemon-n8slw","kube-system/konnectivity-agent-khwpz","openshift-insights/insights-runtime-extractor-rb8r5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal","openshift-image-registry/node-ca-xlsrl","openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8","openshift-multus/multus-additional-cni-plugins-g9pb4","kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal","openshift-dns/dns-default-hwt9g","openshift-ovn-kubernetes/ovnkube-node-vhcwk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn","openshift-network-diagnostics/network-check-target-kjmq9","openshift-dns/node-resolver-mxcvd"] Oct 31 02:55:19.401115 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.401094 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:19.406074 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.406051 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:19.406794 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.406769 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 02:55:19.406896 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.406811 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.407398 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.407377 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.407486 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.407461 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 02:55:19.407555 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.407488 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rkjm6\"" Oct 31 02:55:19.407555 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.407530 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q5ns6\"" Oct 31 02:55:19.407648 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.407569 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sts47\"" Oct 31 02:55:19.408125 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.408106 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 02:55:19.408190 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.408147 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7qc7w\"" Oct 31 02:55:19.408244 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.408228 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ls2x2\"" Oct 31 02:55:19.408285 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.408259 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tbpkr\"" Oct 31 02:55:19.408330 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.408320 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 02:55:19.409293 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.409272 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 02:55:19.409404 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.409310 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wcv7f\"" Oct 31 02:55:19.409404 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.409339 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x5rg4\"" Oct 31 02:55:19.409932 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.409912 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wr2d\"" Oct 31 02:55:19.410010 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.409973 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-49z5f\"" Oct 31 02:55:19.410082 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410012 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.410082 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410021 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 02:55:19.410082 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410048 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jvhtj\"" Oct 31 02:55:19.410082 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410056 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 02:55:19.410082 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410065 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.410082 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410018 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 02:55:19.410388 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410376 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 02:55:19.410418 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410411 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 02:55:19.410449 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410442 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.410545 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410492 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.410545 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410499 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.410545 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410509 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410561 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410572 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410579 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410574 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410595 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410613 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410619 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410622 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410614 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410647 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410614 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 02:55:19.410687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410626 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.410994 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.410843 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 02:55:19.412398 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412380 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 02:55:19.412460 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412423 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jsfq8\"" Oct 31 02:55:19.412460 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412439 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 02:55:19.412460 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412442 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.412581 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412472 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.412581 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412493 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2b67f\"" Oct 31 02:55:19.412667 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412595 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x8dcs\"" Oct 31 02:55:19.412714 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412675 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sgkvt\"" Oct 31 02:55:19.412775 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412763 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 02:55:19.412828 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412811 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hnld2\"" Oct 31 02:55:19.412828 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412818 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 02:55:19.412927 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412846 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 02:55:19.412927 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412871 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 02:55:19.412927 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412889 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.412927 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412910 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.413081 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412953 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 02:55:19.413081 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.412960 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:19.417361 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417336 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:19.417449 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417364 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fd7955b-02fe-411e-83a4-abac6fdbb242-host-slash\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:55:19.417449 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417379 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-systemd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417449 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417393 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-etc-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417449 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417409 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-bin\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417635 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417459 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-config\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417635 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417496 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdx4z\" (UniqueName: \"kubernetes.io/projected/6fd7955b-02fe-411e-83a4-abac6fdbb242-kube-api-access-zdx4z\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:55:19.417635 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417520 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovn-node-metrics-cert\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417635 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417546 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-systemd-units\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417635 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417572 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-slash\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417641 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417675 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-env-overrides\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417702 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee641d86-9b04-4f02-a22b-7cc55694a018-hosts-file\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:55:19.417847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417727 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-netns\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417739 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovn-node-metrics-cert\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417758 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-config\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417769 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-var-lib-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.417847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417778 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-original-pull-secret\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:19.417847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417801 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-ovn\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.418151 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417851 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-script-lib\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.418151 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417900 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-dbus\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:19.418151 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417908 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-env-overrides\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.418151 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417919 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-node-log\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.418151 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.417938 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4vt\" (UniqueName: \"kubernetes.io/projected/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-kube-api-access-lx4vt\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.418151 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418065 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fd7955b-02fe-411e-83a4-abac6fdbb242-iptables-alerter-script\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:55:19.418151 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418088 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-ovnkube-script-lib\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.418151 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418136 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee641d86-9b04-4f02-a22b-7cc55694a018-tmp-dir\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:55:19.418350 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418182 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.418350 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418231 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee641d86-9b04-4f02-a22b-7cc55694a018-tmp-dir\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:55:19.418350 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418233 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fd7955b-02fe-411e-83a4-abac6fdbb242-iptables-alerter-script\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:55:19.418350 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418271 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwx7\" (UniqueName: \"kubernetes.io/projected/ee641d86-9b04-4f02-a22b-7cc55694a018-kube-api-access-9jwx7\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:55:19.418350 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418304 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-kubelet-config\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:19.418350 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418325 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-log-socket\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.418526 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418359 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-kubelet\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.418526 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418415 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-netd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.418526 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.418441 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.428798 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.428768 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4vt\" (UniqueName: \"kubernetes.io/projected/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-kube-api-access-lx4vt\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.428880 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.428771 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwx7\" (UniqueName: \"kubernetes.io/projected/ee641d86-9b04-4f02-a22b-7cc55694a018-kube-api-access-9jwx7\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:55:19.428880 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.428774 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdx4z\" (UniqueName: \"kubernetes.io/projected/6fd7955b-02fe-411e-83a4-abac6fdbb242-kube-api-access-zdx4z\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:55:19.462876 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.462855 4456 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:19.462974 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.462960 4456 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:19.476146 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.476129 4456 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:19.476199 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:19.476184 4456 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:19.476555 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.476541 4456 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:19.476604 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:19.476571 4456 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-107.us-east-2.compute.internal" Oct 31 02:55:19.500914 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.500896 4456 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 02:55:19.519586 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519567 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:19.519639 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519593 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-system-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.519639 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519609 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-kubelet\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.519639 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519626 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4bb32ff-188e-4471-8c62-08fae7e26c8c-cert\") pod \"ingress-canary-cqgsq\" (UID: \"b4bb32ff-188e-4471-8c62-08fae7e26c8c\") " pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:19.519746 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519647 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-kubelet\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.519746 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519684 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-kubelet\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.519746 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519684 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-netd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.519746 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519723 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.519898 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519793 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-netd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.519898 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519816 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-249fn\" (UniqueName: \"kubernetes.io/projected/4249fb31-38de-4684-abd0-6d49c6dacd59-kube-api-access-249fn\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.519898 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519836 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysconfig\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.519898 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519857 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-tuned\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.519898 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519877 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-socket-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.520079 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519939 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4bb32ff-188e-4471-8c62-08fae7e26c8c-cert\") pod \"ingress-canary-cqgsq\" (UID: \"b4bb32ff-188e-4471-8c62-08fae7e26c8c\") " pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:19.520079 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519968 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fd7955b-02fe-411e-83a4-abac6fdbb242-host-slash\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:55:19.520079 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.519999 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-tuned\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.520079 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520072 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fd7955b-02fe-411e-83a4-abac6fdbb242-host-slash\") pod \"iptables-alerter-tq4b7\" (UID: \"6fd7955b-02fe-411e-83a4-abac6fdbb242\") " pod="openshift-network-operator/iptables-alerter-tq4b7" Oct 31 02:55:19.520079 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520073 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ded5081-b545-47f2-be60-975cf616f05f-crio-socket\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.520231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520105 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.520231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520145 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.520231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520164 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-lib-modules\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.520231 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520206 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-hostroot\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.520357 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520238 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-multus-daemon-config\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.520357 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520266 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxm7m\" (UniqueName: \"kubernetes.io/projected/2be3a160-6ada-4e51-a6bf-19745a8670f4-kube-api-access-jxm7m\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:55:19.520357 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520322 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-systemd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.520357 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520323 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ded5081-b545-47f2-be60-975cf616f05f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.520472 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520427 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-etc-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.520805 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520771 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-systemd\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.520805 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520798 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-etc-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.520913 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520809 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-multus-daemon-config\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.520983 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.520796 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-system-cni-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.521135 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.521118 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-cnibin\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.521238 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.521222 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.521327 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.521315 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-systemd\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.521409 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.521395 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-host\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.521497 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.521486 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-tmp\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.521597 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.521584 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-sys\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.521687 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.521675 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-cni-binary-copy\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.521774 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.521762 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-bin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.521856 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.521844 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.521947 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.521936 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee641d86-9b04-4f02-a22b-7cc55694a018-hosts-file\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:55:19.522063 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522033 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.522166 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522154 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-netns\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.522251 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522240 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-ovn\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.522332 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522319 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-dbus\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:19.522418 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522406 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4ll\" (UniqueName: \"kubernetes.io/projected/b4bb32ff-188e-4471-8c62-08fae7e26c8c-kube-api-access-mv4ll\") pod \"ingress-canary-cqgsq\" (UID: \"b4bb32ff-188e-4471-8c62-08fae7e26c8c\") " pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:19.522612 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522591 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-node-log\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.522688 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522663 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-node-log\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.522768 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522755 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdh6m\" (UniqueName: \"kubernetes.io/projected/4ded5081-b545-47f2-be60-975cf616f05f-kube-api-access-gdh6m\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.522874 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522857 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-ovn\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.522942 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522867 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-netns\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.522942 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522894 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-dbus\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:19.522942 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522929 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0facc8a8-af14-4207-bb9f-2e36129b0343-tmp\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.523142 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.522980 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.523142 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523073 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-modprobe-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.523142 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523094 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79426555-723a-431c-862c-8f936334be71-cni-binary-copy\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.523142 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523114 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:19.523399 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523377 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-agent-certs\") pod \"konnectivity-agent-khwpz\" (UID: \"24d5a9bc-29a3-42b5-bf90-9468b578b0f7\") " pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:55:19.523460 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523423 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.523511 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523473 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pnv\" (UniqueName: \"kubernetes.io/projected/24901959-cd66-4412-a070-aeb1cfaa8444-kube-api-access-g9pnv\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:19.523558 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523511 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-netns\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.523610 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523557 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-etc-kubernetes\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.523610 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523590 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l65r\" (UniqueName: \"kubernetes.io/projected/79426555-723a-431c-862c-8f936334be71-kube-api-access-4l65r\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.523695 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523620 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-registration-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.523695 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523647 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sskj6\" (UniqueName: \"kubernetes.io/projected/af399405-2475-49b9-9fdd-7f43dff044b6-kube-api-access-sskj6\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.523695 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523681 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-kubelet-config\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:19.523825 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523721 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-log-socket\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.523825 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523751 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.523825 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523783 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-konnectivity-ca\") pod \"konnectivity-agent-khwpz\" (UID: \"24d5a9bc-29a3-42b5-bf90-9468b578b0f7\") " pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:55:19.523825 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523817 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-sys-fs\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.523994 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523853 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wsb\" (UniqueName: \"kubernetes.io/projected/88c4d958-b761-4b20-b3dd-c02e87b8b10c-kube-api-access-t4wsb\") pod \"migrator-66df5d496b-xgtr8\" (UID: \"88c4d958-b761-4b20-b3dd-c02e87b8b10c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8" Oct 31 02:55:19.523994 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523881 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/810e723f-fc3a-4477-a30d-dc086bd338b1-config-volume\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:19.523994 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523883 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-agent-certs\") pod \"konnectivity-agent-khwpz\" (UID: \"24d5a9bc-29a3-42b5-bf90-9468b578b0f7\") " pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:55:19.523994 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523382 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee641d86-9b04-4f02-a22b-7cc55694a018-hosts-file\") pod \"node-resolver-mxcvd\" (UID: \"ee641d86-9b04-4f02-a22b-7cc55694a018\") " pod="openshift-dns/node-resolver-mxcvd" Oct 31 02:55:19.523994 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523957 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-run-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.523994 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523972 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7047af-e1c4-4e1c-81ea-f777c15b0d97-kubelet-config\") pod \"global-pull-secret-syncer-vtlr6\" (UID: \"3b7047af-e1c4-4e1c-81ea-f777c15b0d97\") " pod="kube-system/global-pull-secret-syncer-vtlr6" Oct 31 02:55:19.524293 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.523994 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-log-socket\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.524293 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.524010 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-k8s-cni-cncf-io\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.524293 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.524107 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-bin\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.524293 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.524191 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24d5a9bc-29a3-42b5-bf90-9468b578b0f7-konnectivity-ca\") pod \"konnectivity-agent-khwpz\" (UID: \"24d5a9bc-29a3-42b5-bf90-9468b578b0f7\") " pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:55:19.524293 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.524265 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-var-lib-kubelet\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.524485 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.524411 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/810e723f-fc3a-4477-a30d-dc086bd338b1-config-volume\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:19.524578 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.524561 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.524578 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.524567 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.524696 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.524667 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-multus\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.524955 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.524929 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-cni-bin\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.525028 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525006 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:19.525105 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525083 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-kubernetes\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.525138 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525107 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58487\" (UniqueName: \"kubernetes.io/projected/0facc8a8-af14-4207-bb9f-2e36129b0343-kube-api-access-58487\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.525166 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525123 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-socket-dir-parent\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.525166 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525155 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-multus-certs\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.525232 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525169 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2be3a160-6ada-4e51-a6bf-19745a8670f4-host\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:55:19.525232 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525184 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-systemd-units\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.525232 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525199 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-slash\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.525322 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525248 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-host-slash\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.525322 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525284 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-systemd-units\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.525322 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525316 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.525424 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525360 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.525424 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525393 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-device-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.525496 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525423 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2be3a160-6ada-4e51-a6bf-19745a8670f4-serviceca\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:55:19.525496 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525446 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-var-lib-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.525560 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525542 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.525589 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525562 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.525589 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525550 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c05a3117-b24e-4810-9a49-2dd3cc9cc88b-var-lib-openvswitch\") pod \"ovnkube-node-vhcwk\" (UID: \"c05a3117-b24e-4810-9a49-2dd3cc9cc88b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:19.525655 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525603 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/810e723f-fc3a-4477-a30d-dc086bd338b1-tmp-dir\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:19.525655 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525629 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ded5081-b545-47f2-be60-975cf616f05f-data-volume\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.525741 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525655 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-conf\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.525741 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525668 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2be3a160-6ada-4e51-a6bf-19745a8670f4-serviceca\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:55:19.525741 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525699 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-cnibin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.525741 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525702 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/810e723f-fc3a-4477-a30d-dc086bd338b1-tmp-dir\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:19.525741 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525724 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ded5081-b545-47f2-be60-975cf616f05f-data-volume\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.525886 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525743 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvb7x\" (UniqueName: \"kubernetes.io/projected/810e723f-fc3a-4477-a30d-dc086bd338b1-kube-api-access-jvb7x\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:19.525886 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525772 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-run\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.525886 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525797 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-conf-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.525886 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525827 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.525886 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525838 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.526020 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.525968 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4249fb31-38de-4684-abd0-6d49c6dacd59-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.531391 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.531370 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-249fn\" (UniqueName: \"kubernetes.io/projected/4249fb31-38de-4684-abd0-6d49c6dacd59-kube-api-access-249fn\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.531531 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.531515 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxm7m\" (UniqueName: \"kubernetes.io/projected/2be3a160-6ada-4e51-a6bf-19745a8670f4-kube-api-access-jxm7m\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:55:19.536563 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.536544 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4ll\" (UniqueName: \"kubernetes.io/projected/b4bb32ff-188e-4471-8c62-08fae7e26c8c-kube-api-access-mv4ll\") pod \"ingress-canary-cqgsq\" (UID: \"b4bb32ff-188e-4471-8c62-08fae7e26c8c\") " pod="openshift-ingress-canary/ingress-canary-cqgsq" Oct 31 02:55:19.538166 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.538131 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sskj6\" (UniqueName: \"kubernetes.io/projected/af399405-2475-49b9-9fdd-7f43dff044b6-kube-api-access-sskj6\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.538288 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.538269 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l65r\" (UniqueName: \"kubernetes.io/projected/79426555-723a-431c-862c-8f936334be71-kube-api-access-4l65r\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.538410 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.538387 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvb7x\" (UniqueName: \"kubernetes.io/projected/810e723f-fc3a-4477-a30d-dc086bd338b1-kube-api-access-jvb7x\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:19.538664 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.538648 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58487\" (UniqueName: \"kubernetes.io/projected/0facc8a8-af14-4207-bb9f-2e36129b0343-kube-api-access-58487\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.538783 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.538761 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wsb\" (UniqueName: \"kubernetes.io/projected/88c4d958-b761-4b20-b3dd-c02e87b8b10c-kube-api-access-t4wsb\") pod \"migrator-66df5d496b-xgtr8\" (UID: \"88c4d958-b761-4b20-b3dd-c02e87b8b10c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgtr8" Oct 31 02:55:19.538832 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.538778 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9pnv\" (UniqueName: \"kubernetes.io/projected/24901959-cd66-4412-a070-aeb1cfaa8444-kube-api-access-g9pnv\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:19.538992 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.538976 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnm92\" (UniqueName: \"kubernetes.io/projected/e02f7dec-7aaf-4418-b520-74ce9c2ebbf0-kube-api-access-vnm92\") pod \"network-check-target-kjmq9\" (UID: \"e02f7dec-7aaf-4418-b520-74ce9c2ebbf0\") " pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:19.540931 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.540911 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdh6m\" (UniqueName: \"kubernetes.io/projected/4ded5081-b545-47f2-be60-975cf616f05f-kube-api-access-gdh6m\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.627027 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627000 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-cnibin\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.627027 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627026 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627176 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627056 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-systemd\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627176 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627081 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-host\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627176 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627098 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-cnibin\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.627176 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627119 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-sys\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627176 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627143 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-bin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627182 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627196 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-host\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627204 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-systemd\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627212 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-modprobe-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627216 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-sys\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627216 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627234 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-bin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627247 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627251 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627310 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-modprobe-d\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627355 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:19.627331 4456 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:19.627401 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs podName:24901959-cd66-4412-a070-aeb1cfaa8444 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:20.127383268 +0000 UTC m=+1.807909398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs") pod "network-metrics-daemon-n8slw" (UID: "24901959-cd66-4412-a070-aeb1cfaa8444") : secret "metrics-daemon-secret" not found Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627432 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-netns\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627448 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-etc-kubernetes\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627463 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-registration-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627479 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-sys-fs\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627494 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-k8s-cni-cncf-io\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627521 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-var-lib-kubelet\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627526 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-etc-kubernetes\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627554 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-registration-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627555 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627563 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-netns\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627575 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-k8s-cni-cncf-io\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627554 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-sys-fs\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627584 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-multus\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627603 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627616 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-kubernetes\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627616 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-cni-multus\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627646 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-socket-dir-parent\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627572 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-var-lib-kubelet\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627671 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-multus-certs\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627679 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-kubernetes\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627693 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2be3a160-6ada-4e51-a6bf-19745a8670f4-host\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627704 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-socket-dir-parent\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627710 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-run-multus-certs\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.627715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627719 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627739 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2be3a160-6ada-4e51-a6bf-19745a8670f4-host\") pod \"node-ca-xlsrl\" (UID: \"2be3a160-6ada-4e51-a6bf-19745a8670f4\") " pod="openshift-image-registry/node-ca-xlsrl" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627740 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-device-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627756 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627768 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-conf\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627793 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-cnibin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627815 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-run\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627819 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-device-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627830 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-conf-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627856 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-multus-conf-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627857 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-cnibin\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627871 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-run\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627874 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysctl-conf\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627885 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627913 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-system-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:19.627932 4456 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627936 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-kubelet\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627963 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627975 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-system-cni-dir\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.627985 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysconfig\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628007 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-host-var-lib-kubelet\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628014 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-socket-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628027 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628055 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-etc-sysconfig\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628056 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ded5081-b545-47f2-be60-975cf616f05f-crio-socket\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628057 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af399405-2475-49b9-9fdd-7f43dff044b6-socket-dir\") pod \"aws-ebs-csi-driver-node-drvwn\" (UID: \"af399405-2475-49b9-9fdd-7f43dff044b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drvwn" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:19.628068 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls podName:810e723f-fc3a-4477-a30d-dc086bd338b1 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:20.128052431 +0000 UTC m=+1.808578542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls") pod "dns-default-hwt9g" (UID: "810e723f-fc3a-4477-a30d-dc086bd338b1") : secret "dns-default-metrics-tls" not found Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628095 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ded5081-b545-47f2-be60-975cf616f05f-crio-socket\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628108 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628138 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-lib-modules\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:19.628201 4456 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628222 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0facc8a8-af14-4207-bb9f-2e36129b0343-lib-modules\") pod \"tuned-ctx6n\" (UID: \"0facc8a8-af14-4207-bb9f-2e36129b0343\") " pod="openshift-cluster-node-tuning-operator/tuned-ctx6n" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628243 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-hostroot\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:19.628258 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:20.128244259 +0000 UTC m=+1.808770368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628279 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-system-cni-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628281 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79426555-723a-431c-862c-8f936334be71-hostroot\") pod \"multus-qhcg9\" (UID: \"79426555-723a-431c-862c-8f936334be71\") " pod="openshift-multus/multus-qhcg9" Oct 31 02:55:19.628590 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:19.628320 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4249fb31-38de-4684-abd0-6d49c6dacd59-system-cni-dir\") pod \"multus-additional-cni-plugins-g9pb4\" (UID: \"4249fb31-38de-4684-abd0-6d49c6dacd59\") " pod="openshift-multus/multus-additional-cni-plugins-g9pb4" Oct 31 02:55:20.131409 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:20.131320 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:20.131409 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:20.131374 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:20.131409 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:20.131395 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:20.131652 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:20.131480 4456 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 02:55:20.131652 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:20.131498 4456 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:20.131652 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:20.131532 4456 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:20.131652 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:20.131563 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.131549787 +0000 UTC m=+2.812075892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:20.131652 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:20.131586 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls podName:810e723f-fc3a-4477-a30d-dc086bd338b1 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.131568089 +0000 UTC m=+2.812094211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls") pod "dns-default-hwt9g" (UID: "810e723f-fc3a-4477-a30d-dc086bd338b1") : secret "dns-default-metrics-tls" not found Oct 31 02:55:20.131652 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:20.131608 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs podName:24901959-cd66-4412-a070-aeb1cfaa8444 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.131598522 +0000 UTC m=+2.812124628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs") pod "network-metrics-daemon-n8slw" (UID: "24901959-cd66-4412-a070-aeb1cfaa8444") : secret "metrics-daemon-secret" not found Oct 31 02:55:21.137264 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:21.137224 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:21.137264 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:21.137269 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:21.137766 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:21.137359 4456 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:21.137766 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:21.137361 4456 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:21.137766 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:21.137405 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:21.137766 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:21.137415 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:23.137401882 +0000 UTC m=+4.817928003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:21.137766 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:21.137499 4456 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 02:55:21.137766 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:21.137503 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls podName:810e723f-fc3a-4477-a30d-dc086bd338b1 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:23.137463007 +0000 UTC m=+4.817989117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls") pod "dns-default-hwt9g" (UID: "810e723f-fc3a-4477-a30d-dc086bd338b1") : secret "dns-default-metrics-tls" not found Oct 31 02:55:21.137766 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:21.137544 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs podName:24901959-cd66-4412-a070-aeb1cfaa8444 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:23.137534908 +0000 UTC m=+4.818061013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs") pod "network-metrics-daemon-n8slw" (UID: "24901959-cd66-4412-a070-aeb1cfaa8444") : secret "metrics-daemon-secret" not found Oct 31 02:55:22.253298 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:22.253261 4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-khwpz_24d5a9bc-29a3-42b5-bf90-9468b578b0f7/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:22.415009 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:22.414973 4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-khwpz_24d5a9bc-29a3-42b5-bf90-9468b578b0f7/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:22.541631 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:22.541562 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mxcvd_ee641d86-9b04-4f02-a22b-7cc55694a018/dns-node-resolver/0.log" Oct 31 02:55:22.735609 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:22.735581 4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-khwpz_24d5a9bc-29a3-42b5-bf90-9468b578b0f7/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:23.150232 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.150196 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:23.150394 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.150246 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:23.150394 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.150275 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:23.150394 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:23.150361 4456 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:23.150504 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:23.150430 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls podName:810e723f-fc3a-4477-a30d-dc086bd338b1 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:27.150406665 +0000 UTC m=+8.830932773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls") pod "dns-default-hwt9g" (UID: "810e723f-fc3a-4477-a30d-dc086bd338b1") : secret "dns-default-metrics-tls" not found Oct 31 02:55:23.150504 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:23.150363 4456 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:23.150504 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:23.150495 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:27.150482072 +0000 UTC m=+8.831008191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:23.150641 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.150622 4456 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 02:55:23.152592 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.152570 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24901959-cd66-4412-a070-aeb1cfaa8444-metrics-certs\") pod \"network-metrics-daemon-n8slw\" (UID: \"24901959-cd66-4412-a070-aeb1cfaa8444\") " pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:23.176906 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.176878 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:23.302267 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.302225 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8slw" Oct 31 02:55:23.554882 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.554855 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xlsrl_2be3a160-6ada-4e51-a6bf-19745a8670f4/node-ca/0.log" Oct 31 02:55:23.580486 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.580462 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n8slw"] Oct 31 02:55:23.583255 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:23.583209 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24901959_cd66_4412_a070_aeb1cfaa8444.slice/crio-3c986d03a06d2aeb36e96ce03ff2e6f8210d0cb7d1c4df4f0d53cc5997201623 WatchSource:0}: Error finding container 3c986d03a06d2aeb36e96ce03ff2e6f8210d0cb7d1c4df4f0d53cc5997201623: Status 404 returned error can't find the container with id 3c986d03a06d2aeb36e96ce03ff2e6f8210d0cb7d1c4df4f0d53cc5997201623 Oct 31 02:55:23.585294 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.585277 4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 02:55:23.941879 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:23.941849 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cqgsq_b4bb32ff-188e-4471-8c62-08fae7e26c8c/serve-healthcheck-canary/0.log" Oct 31 02:55:24.476003 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:24.475971 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n8slw" event={"ID":"24901959-cd66-4412-a070-aeb1cfaa8444","Type":"ContainerStarted","Data":"3c986d03a06d2aeb36e96ce03ff2e6f8210d0cb7d1c4df4f0d53cc5997201623"} Oct 31 02:55:24.749228 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:24.749153 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-xgtr8_88c4d958-b761-4b20-b3dd-c02e87b8b10c/migrator/0.log" Oct 31 02:55:25.479859 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:25.479826 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n8slw" event={"ID":"24901959-cd66-4412-a070-aeb1cfaa8444","Type":"ContainerStarted","Data":"51092eaa2d96970aab6f354544de3a373402ed3516579204373ca6d3a781c7db"} Oct 31 02:55:25.480373 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:25.479884 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n8slw" event={"ID":"24901959-cd66-4412-a070-aeb1cfaa8444","Type":"ContainerStarted","Data":"e9a0746ec3e9267cb52c1ffcdb0f73e45269c8f33fd257776294a0a5a56d6b17"} Oct 31 02:55:27.180750 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:27.180715 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:27.180750 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:27.180764 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:27.181269 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:27.180871 4456 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:27.181269 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:27.180896 4456 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:27.181269 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:27.180935 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls podName:810e723f-fc3a-4477-a30d-dc086bd338b1 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:35.180918912 +0000 UTC m=+16.861445034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls") pod "dns-default-hwt9g" (UID: "810e723f-fc3a-4477-a30d-dc086bd338b1") : secret "dns-default-metrics-tls" not found Oct 31 02:55:27.181269 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:55:27.180986 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls podName:4ded5081-b545-47f2-be60-975cf616f05f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:35.180968805 +0000 UTC m=+16.861494934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb8r5" (UID: "4ded5081-b545-47f2-be60-975cf616f05f") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:27.455574 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:27.455480 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:27.455744 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:27.455669 4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:29.545658 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:29.545612 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:30.439314 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:30.439285 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-khwpz" Oct 31 02:55:32.395058 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:32.395008 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhcwk" Oct 31 02:55:35.235343 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:35.235302 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:35.235854 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:35.235357 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:35.237642 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:35.237614 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/810e723f-fc3a-4477-a30d-dc086bd338b1-metrics-tls\") pod \"dns-default-hwt9g\" (UID: \"810e723f-fc3a-4477-a30d-dc086bd338b1\") " pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:35.237743 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:35.237660 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ded5081-b545-47f2-be60-975cf616f05f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb8r5\" (UID: \"4ded5081-b545-47f2-be60-975cf616f05f\") " pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:35.306900 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:35.306874 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:35.307972 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:35.307954 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rb8r5" Oct 31 02:55:35.580797 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:35.580775 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hwt9g"] Oct 31 02:55:35.583399 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:35.583371 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810e723f_fc3a_4477_a30d_dc086bd338b1.slice/crio-3618f283c4a915fa055df22d806d0ee7c32ef9cd5fc6462f204521bd60672884 WatchSource:0}: Error finding container 3618f283c4a915fa055df22d806d0ee7c32ef9cd5fc6462f204521bd60672884: Status 404 returned error can't find the container with id 3618f283c4a915fa055df22d806d0ee7c32ef9cd5fc6462f204521bd60672884 Oct 31 02:55:35.583681 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:35.583662 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rb8r5"] Oct 31 02:55:35.586382 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:35.586361 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ded5081_b545_47f2_be60_975cf616f05f.slice/crio-9ab72789eb152988dae808a5ced8b7ad19ef1708c0fa4047ed9ca801d3e29c68 WatchSource:0}: Error finding container 9ab72789eb152988dae808a5ced8b7ad19ef1708c0fa4047ed9ca801d3e29c68: Status 404 returned error can't find the container with id 9ab72789eb152988dae808a5ced8b7ad19ef1708c0fa4047ed9ca801d3e29c68 Oct 31 02:55:36.508754 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:36.508372 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hwt9g" event={"ID":"810e723f-fc3a-4477-a30d-dc086bd338b1","Type":"ContainerStarted","Data":"3618f283c4a915fa055df22d806d0ee7c32ef9cd5fc6462f204521bd60672884"} Oct 31 02:55:36.510570 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:36.510543 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rb8r5" event={"ID":"4ded5081-b545-47f2-be60-975cf616f05f","Type":"ContainerStarted","Data":"eff1d8e541cf36146cac4c680a6967728b24cf1c0ab8f41d06ea6791734054d3"} Oct 31 02:55:36.510673 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:36.510599 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rb8r5" event={"ID":"4ded5081-b545-47f2-be60-975cf616f05f","Type":"ContainerStarted","Data":"bddb27add13e49a9a4a9bd1fb8b6e10f91b6ebc4980338ed3438fcb988478004"} Oct 31 02:55:36.510673 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:36.510626 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rb8r5" event={"ID":"4ded5081-b545-47f2-be60-975cf616f05f","Type":"ContainerStarted","Data":"9ab72789eb152988dae808a5ced8b7ad19ef1708c0fa4047ed9ca801d3e29c68"} Oct 31 02:55:37.520190 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:37.520160 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hwt9g" event={"ID":"810e723f-fc3a-4477-a30d-dc086bd338b1","Type":"ContainerStarted","Data":"b9f329980a746de0b71c59b2ae4fb8834cd955764505eeae7c63bddb043a9475"} Oct 31 02:55:38.524333 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:38.524297 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hwt9g" event={"ID":"810e723f-fc3a-4477-a30d-dc086bd338b1","Type":"ContainerStarted","Data":"17af055e9001fc3f43351dd94e3d26c2598b587abc9f6e8efb3181eec6f3e267"} Oct 31 02:55:38.524762 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:38.524403 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:38.526260 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:38.526235 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rb8r5" event={"ID":"4ded5081-b545-47f2-be60-975cf616f05f","Type":"ContainerStarted","Data":"7e5f4e7dcbcb8a8d2297e475c8e6c88d60c06d77e01c2ffe54e9430a97355140"} Oct 31 02:55:39.041094 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:39.041057 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj"] Oct 31 02:55:39.044211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:39.044196 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj" Oct 31 02:55:39.056675 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:39.056650 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj"] Oct 31 02:55:39.058105 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:39.058082 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 02:55:39.058105 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:39.058092 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ssg9d\"" Oct 31 02:55:39.158654 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:39.158625 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b7cbf4db-e402-4071-bdaa-7c4ffc6d3626-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lmdmj\" (UID: \"b7cbf4db-e402-4071-bdaa-7c4ffc6d3626\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj" Oct 31 02:55:39.259735 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:39.259703 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b7cbf4db-e402-4071-bdaa-7c4ffc6d3626-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lmdmj\" (UID: \"b7cbf4db-e402-4071-bdaa-7c4ffc6d3626\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj" Oct 31 02:55:39.261975 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:39.261956 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b7cbf4db-e402-4071-bdaa-7c4ffc6d3626-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lmdmj\" (UID: \"b7cbf4db-e402-4071-bdaa-7c4ffc6d3626\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj" Oct 31 02:55:39.355012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:39.354931 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj" Oct 31 02:55:39.628906 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:39.628887 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj"] Oct 31 02:55:39.630744 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:39.630720 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cbf4db_e402_4071_bdaa_7c4ffc6d3626.slice/crio-d6389ff2cf4cb6acdf8483f6401aebe17522596ea6183f3bbfc1c3f4c2a582c1 WatchSource:0}: Error finding container d6389ff2cf4cb6acdf8483f6401aebe17522596ea6183f3bbfc1c3f4c2a582c1: Status 404 returned error can't find the container with id d6389ff2cf4cb6acdf8483f6401aebe17522596ea6183f3bbfc1c3f4c2a582c1 Oct 31 02:55:40.534194 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:40.534160 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj" event={"ID":"b7cbf4db-e402-4071-bdaa-7c4ffc6d3626","Type":"ContainerStarted","Data":"d6389ff2cf4cb6acdf8483f6401aebe17522596ea6183f3bbfc1c3f4c2a582c1"} Oct 31 02:55:41.538462 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.538428 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj" event={"ID":"b7cbf4db-e402-4071-bdaa-7c4ffc6d3626","Type":"ContainerStarted","Data":"84a38d324f43e6b299c549c47239601b9c14fed600fe537670b80b20eedd53f0"} Oct 31 02:55:41.539092 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.538634 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj" Oct 31 02:55:41.543096 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.543073 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj" Oct 31 02:55:41.559125 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.559085 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lmdmj" podStartSLOduration=1.454947313 podStartE2EDuration="2.559074818s" totalImagesPullingTime="1.104127505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:41.558648726 +0000 UTC m=+23.239174854" watchObservedRunningTime="2026-10-31 02:55:41.559074818 +0000 UTC m=+23.239600936" Oct 31 02:55:41.899996 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.899919 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ft948"] Oct 31 02:55:41.903004 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.902989 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:41.911786 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.911765 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 02:55:41.914634 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.914611 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ft948"] Oct 31 02:55:41.915790 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.915767 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 02:55:41.916122 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.916107 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 02:55:41.916173 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.916136 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 02:55:41.931996 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.931971 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-p8dqb\"" Oct 31 02:55:41.932114 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.932025 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 02:55:41.978480 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.978453 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b54ae00b-53b0-426d-b12a-88a8c03dfc13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:41.978569 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.978496 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b54ae00b-53b0-426d-b12a-88a8c03dfc13-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:41.978569 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.978521 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b54ae00b-53b0-426d-b12a-88a8c03dfc13-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:41.978653 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:41.978618 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2488\" (UniqueName: \"kubernetes.io/projected/b54ae00b-53b0-426d-b12a-88a8c03dfc13-kube-api-access-c2488\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:42.079712 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.079680 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b54ae00b-53b0-426d-b12a-88a8c03dfc13-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:42.079712 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.079713 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b54ae00b-53b0-426d-b12a-88a8c03dfc13-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:42.079897 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.079742 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2488\" (UniqueName: \"kubernetes.io/projected/b54ae00b-53b0-426d-b12a-88a8c03dfc13-kube-api-access-c2488\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:42.079897 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.079779 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b54ae00b-53b0-426d-b12a-88a8c03dfc13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:42.080423 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.080386 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b54ae00b-53b0-426d-b12a-88a8c03dfc13-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:42.082197 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.082178 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b54ae00b-53b0-426d-b12a-88a8c03dfc13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:42.082363 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.082340 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b54ae00b-53b0-426d-b12a-88a8c03dfc13-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:42.093387 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.093359 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2488\" (UniqueName: \"kubernetes.io/projected/b54ae00b-53b0-426d-b12a-88a8c03dfc13-kube-api-access-c2488\") pod \"prometheus-operator-9cd89b74c-ft948\" (UID: \"b54ae00b-53b0-426d-b12a-88a8c03dfc13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:42.215824 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.215786 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" Oct 31 02:55:42.484938 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.484916 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ft948"] Oct 31 02:55:42.487410 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:42.487382 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54ae00b_53b0_426d_b12a_88a8c03dfc13.slice/crio-afbbf37c036b3e33ad6dabb239b59e917848115c61506a4aa7be18e7d6b242fb WatchSource:0}: Error finding container afbbf37c036b3e33ad6dabb239b59e917848115c61506a4aa7be18e7d6b242fb: Status 404 returned error can't find the container with id afbbf37c036b3e33ad6dabb239b59e917848115c61506a4aa7be18e7d6b242fb Oct 31 02:55:42.541977 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:42.541947 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" event={"ID":"b54ae00b-53b0-426d-b12a-88a8c03dfc13","Type":"ContainerStarted","Data":"afbbf37c036b3e33ad6dabb239b59e917848115c61506a4aa7be18e7d6b242fb"} Oct 31 02:55:44.550385 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:44.550350 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" event={"ID":"b54ae00b-53b0-426d-b12a-88a8c03dfc13","Type":"ContainerStarted","Data":"98994a09d2a622dcd89b295a2c0a1d0dde407632208824a622ee1c7731ee42c4"} Oct 31 02:55:44.550753 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:44.550401 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" event={"ID":"b54ae00b-53b0-426d-b12a-88a8c03dfc13","Type":"ContainerStarted","Data":"4e604555d5717258656380288e637432883c49472074d33bc9970a92e9785fb8"} Oct 31 02:55:44.577544 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:44.577500 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ft948" podStartSLOduration=2.378562269 podStartE2EDuration="3.57748823s" totalImagesPullingTime="1.198925961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:44.574679631 +0000 UTC m=+26.255205759" watchObservedRunningTime="2026-10-31 02:55:44.57748823 +0000 UTC m=+26.258014357" Oct 31 02:55:46.271329 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.271296 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k"] Oct 31 02:55:46.274575 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.274553 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.289017 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.288989 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pdt5m\"" Oct 31 02:55:46.289017 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.288989 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 02:55:46.289464 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.289450 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 02:55:46.289763 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.289748 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c"] Oct 31 02:55:46.292763 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.292744 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.293580 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.293560 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k"] Oct 31 02:55:46.308423 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.308396 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-75lnw"] Oct 31 02:55:46.311753 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.311734 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c"] Oct 31 02:55:46.311883 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.311840 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.314137 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.314119 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 02:55:46.386938 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.386912 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 02:55:46.387217 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.387199 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 02:55:46.387313 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.387199 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-b9brn\"" Oct 31 02:55:46.387313 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.387227 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 02:55:46.387313 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.387252 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtdnh\"" Oct 31 02:55:46.387426 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.387393 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 02:55:46.411700 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.411674 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.411811 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.411707 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9e6cae46-c89a-458b-93a5-e313e7c01403-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.411811 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.411728 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.411811 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.411746 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-textfile\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.411811 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.411795 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-tls\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.411974 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.411887 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12194edf-58dc-4167-a1e0-8888ccf43cc9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.411974 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.411910 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9c04022-9e9e-4704-986e-ad2ef2993224-sys\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.411974 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.411928 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b9c04022-9e9e-4704-986e-ad2ef2993224-root\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.411974 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.411945 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e6cae46-c89a-458b-93a5-e313e7c01403-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.412200 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.412011 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.412200 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.412083 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9c04022-9e9e-4704-986e-ad2ef2993224-metrics-client-ca\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.412200 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.412115 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.412349 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.412196 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-accelerators-collector-config\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.412349 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.412232 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5hk\" (UniqueName: \"kubernetes.io/projected/b9c04022-9e9e-4704-986e-ad2ef2993224-kube-api-access-bn5hk\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.412349 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.412258 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12194edf-58dc-4167-a1e0-8888ccf43cc9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.412349 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.412293 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12194edf-58dc-4167-a1e0-8888ccf43cc9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.412349 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.412332 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-wtmp\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.412524 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.412352 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5hh\" (UniqueName: \"kubernetes.io/projected/12194edf-58dc-4167-a1e0-8888ccf43cc9-kube-api-access-wj5hh\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.412524 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.412370 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9j52\" (UniqueName: \"kubernetes.io/projected/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-api-access-b9j52\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.419635 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.419616 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 02:55:46.512828 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.512804 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.512959 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.512844 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9e6cae46-c89a-458b-93a5-e313e7c01403-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.512959 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.512875 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.513109 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.512957 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-textfile\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.513109 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.512986 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-tls\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.513109 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513011 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12194edf-58dc-4167-a1e0-8888ccf43cc9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.513109 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513059 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9c04022-9e9e-4704-986e-ad2ef2993224-sys\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.513109 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513087 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b9c04022-9e9e-4704-986e-ad2ef2993224-root\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.513360 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513114 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e6cae46-c89a-458b-93a5-e313e7c01403-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.513360 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513149 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.513360 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513163 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b9c04022-9e9e-4704-986e-ad2ef2993224-root\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.513360 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513180 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9c04022-9e9e-4704-986e-ad2ef2993224-metrics-client-ca\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.513360 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513224 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.513360 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513270 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-accelerators-collector-config\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.513360 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513304 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5hk\" (UniqueName: \"kubernetes.io/projected/b9c04022-9e9e-4704-986e-ad2ef2993224-kube-api-access-bn5hk\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.513360 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513334 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12194edf-58dc-4167-a1e0-8888ccf43cc9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.513360 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513347 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-textfile\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.514097 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513345 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9e6cae46-c89a-458b-93a5-e313e7c01403-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.514097 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.513712 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9c04022-9e9e-4704-986e-ad2ef2993224-sys\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.514228 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.514192 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-accelerators-collector-config\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.514286 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.514245 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9c04022-9e9e-4704-986e-ad2ef2993224-metrics-client-ca\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.514339 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.514294 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12194edf-58dc-4167-a1e0-8888ccf43cc9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.514339 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.514325 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-wtmp\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.514431 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.514351 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5hh\" (UniqueName: \"kubernetes.io/projected/12194edf-58dc-4167-a1e0-8888ccf43cc9-kube-api-access-wj5hh\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.514431 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.514379 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9j52\" (UniqueName: \"kubernetes.io/projected/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-api-access-b9j52\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.514565 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.514540 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12194edf-58dc-4167-a1e0-8888ccf43cc9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.514798 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.514757 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e6cae46-c89a-458b-93a5-e313e7c01403-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.514884 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.514801 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-wtmp\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.514884 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.514820 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.515747 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.515711 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.515956 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.515935 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9c04022-9e9e-4704-986e-ad2ef2993224-node-exporter-tls\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.516069 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.515949 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.516960 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.516940 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.516960 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.516952 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12194edf-58dc-4167-a1e0-8888ccf43cc9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.517106 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.517062 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12194edf-58dc-4167-a1e0-8888ccf43cc9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.529052 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.528999 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5hk\" (UniqueName: \"kubernetes.io/projected/b9c04022-9e9e-4704-986e-ad2ef2993224-kube-api-access-bn5hk\") pod \"node-exporter-75lnw\" (UID: \"b9c04022-9e9e-4704-986e-ad2ef2993224\") " pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.534830 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.534806 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9j52\" (UniqueName: \"kubernetes.io/projected/9e6cae46-c89a-458b-93a5-e313e7c01403-kube-api-access-b9j52\") pod \"kube-state-metrics-5dc8d7d89b-hrd9c\" (UID: \"9e6cae46-c89a-458b-93a5-e313e7c01403\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.535412 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.535395 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5hh\" (UniqueName: \"kubernetes.io/projected/12194edf-58dc-4167-a1e0-8888ccf43cc9-kube-api-access-wj5hh\") pod \"openshift-state-metrics-7bd7489957-hzk4k\" (UID: \"12194edf-58dc-4167-a1e0-8888ccf43cc9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.589995 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.589973 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" Oct 31 02:55:46.608560 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.608534 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" Oct 31 02:55:46.627918 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.627888 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-75lnw" Oct 31 02:55:46.639022 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:46.638521 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c04022_9e9e_4704_986e_ad2ef2993224.slice/crio-1fd27a44235f5da0e9d08e832979ac878930f0b5cd128ba66c4a70e7eac3823e WatchSource:0}: Error finding container 1fd27a44235f5da0e9d08e832979ac878930f0b5cd128ba66c4a70e7eac3823e: Status 404 returned error can't find the container with id 1fd27a44235f5da0e9d08e832979ac878930f0b5cd128ba66c4a70e7eac3823e Oct 31 02:55:46.866321 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.866292 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k"] Oct 31 02:55:46.868483 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:46.868456 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12194edf_58dc_4167_a1e0_8888ccf43cc9.slice/crio-4881fc03e49c6ca38c85b39161c2d177128fab7bc34ce9080c3f1f2f35084fbc WatchSource:0}: Error finding container 4881fc03e49c6ca38c85b39161c2d177128fab7bc34ce9080c3f1f2f35084fbc: Status 404 returned error can't find the container with id 4881fc03e49c6ca38c85b39161c2d177128fab7bc34ce9080c3f1f2f35084fbc Oct 31 02:55:46.903643 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:46.903619 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c"] Oct 31 02:55:46.905259 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:46.905237 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6cae46_c89a_458b_93a5_e313e7c01403.slice/crio-bfdc58631f563894012b3b7bfa3ed61bae32c03f4e96eb6119e71dbfb0097800 WatchSource:0}: Error finding container bfdc58631f563894012b3b7bfa3ed61bae32c03f4e96eb6119e71dbfb0097800: Status 404 returned error can't find the container with id bfdc58631f563894012b3b7bfa3ed61bae32c03f4e96eb6119e71dbfb0097800 Oct 31 02:55:47.355090 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.355052 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 02:55:47.360082 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.360052 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.365533 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.365501 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 02:55:47.365662 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.365533 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 02:55:47.365662 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.365555 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 02:55:47.365662 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.365586 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 02:55:47.365662 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.365591 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 02:55:47.365662 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.365641 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 02:55:47.365899 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.365758 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 02:55:47.365899 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.365766 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 02:55:47.365899 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.365763 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 02:55:47.365899 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.365813 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wv4jz\"" Oct 31 02:55:47.375538 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.375508 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 02:55:47.420659 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.420629 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.420819 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.420667 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.420819 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.420773 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-web-config\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.420921 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.420848 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsttz\" (UniqueName: \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-kube-api-access-nsttz\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.420921 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.420874 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-config-volume\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.420921 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.420894 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-tls-assets\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.421118 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.421020 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.421118 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.421067 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.421118 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.421087 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.421276 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.421170 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-config-out\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.421276 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.421201 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.421276 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.421235 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.421438 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.421302 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523425 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523388 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523602 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523438 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523602 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523475 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-web-config\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523602 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523537 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsttz\" (UniqueName: \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-kube-api-access-nsttz\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523602 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523566 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-config-volume\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523602 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523593 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-tls-assets\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523866 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523633 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523866 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523661 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523866 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523684 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523866 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523712 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-config-out\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523866 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523739 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523866 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523773 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.523866 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.523798 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.524617 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.524591 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.526581 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.526211 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.526581 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.526406 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.531721 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.531696 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.534861 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.534838 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-web-config\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.535417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.535230 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.535417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.535325 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-config-volume\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.535417 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.535375 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.535666 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.535644 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.536473 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.536427 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.536698 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.536680 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-tls-assets\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.537584 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.537558 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-config-out\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.545769 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.545745 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsttz\" (UniqueName: \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-kube-api-access-nsttz\") pod \"alertmanager-main-0\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:47.563423 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.563351 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" event={"ID":"9e6cae46-c89a-458b-93a5-e313e7c01403","Type":"ContainerStarted","Data":"bfdc58631f563894012b3b7bfa3ed61bae32c03f4e96eb6119e71dbfb0097800"} Oct 31 02:55:47.564825 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.564796 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75lnw" event={"ID":"b9c04022-9e9e-4704-986e-ad2ef2993224","Type":"ContainerStarted","Data":"1fd27a44235f5da0e9d08e832979ac878930f0b5cd128ba66c4a70e7eac3823e"} Oct 31 02:55:47.567208 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.567025 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" event={"ID":"12194edf-58dc-4167-a1e0-8888ccf43cc9","Type":"ContainerStarted","Data":"043c4a4cc2fcceb8184ab0fd21de4021d8faed734b91e9e94f10b4f2a3a9231b"} Oct 31 02:55:47.567208 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.567097 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" event={"ID":"12194edf-58dc-4167-a1e0-8888ccf43cc9","Type":"ContainerStarted","Data":"547e5bdabbdd2d115942830865a2219575ea073d02e26b2f9feaf51cf478945a"} Oct 31 02:55:47.567208 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.567127 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" event={"ID":"12194edf-58dc-4167-a1e0-8888ccf43cc9","Type":"ContainerStarted","Data":"4881fc03e49c6ca38c85b39161c2d177128fab7bc34ce9080c3f1f2f35084fbc"} Oct 31 02:55:47.774051 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:47.774004 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:55:48.149877 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.149847 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 02:55:48.151235 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:48.151208 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777b7650_dacc_43f8_ad77_a9ae3a593410.slice/crio-04285c4f1bb7c636397e53af87e400736103bec67cf339cfd9d1e86546dc0614 WatchSource:0}: Error finding container 04285c4f1bb7c636397e53af87e400736103bec67cf339cfd9d1e86546dc0614: Status 404 returned error can't find the container with id 04285c4f1bb7c636397e53af87e400736103bec67cf339cfd9d1e86546dc0614 Oct 31 02:55:48.371609 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.371582 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-678874f684-2zvzr"] Oct 31 02:55:48.380396 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.380374 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.388139 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.388112 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 02:55:48.388362 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.388130 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 02:55:48.388362 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.388135 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 02:55:48.388612 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.388586 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 02:55:48.388895 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.388873 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8f2aj4olg7rle\"" Oct 31 02:55:48.389699 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.389679 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-678874f684-2zvzr"] Oct 31 02:55:48.390117 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.390097 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 02:55:48.390382 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.390354 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dn52b\"" Oct 31 02:55:48.531851 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.531816 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hwt9g" Oct 31 02:55:48.532662 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.532635 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.532783 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.532687 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebaf5bff-0da3-496e-8461-914f5f374daf-tmp\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.532826 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.532794 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-grpc-tls\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.532859 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.532828 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.532908 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.532892 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-tls\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.532940 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.532917 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.532980 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.532937 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcqwq\" (UniqueName: \"kubernetes.io/projected/ebaf5bff-0da3-496e-8461-914f5f374daf-kube-api-access-pcqwq\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.532980 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.532959 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.533090 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.533073 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebaf5bff-0da3-496e-8461-914f5f374daf-metrics-client-ca\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.573461 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.573426 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerStarted","Data":"04285c4f1bb7c636397e53af87e400736103bec67cf339cfd9d1e86546dc0614"} Oct 31 02:55:48.575248 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.575218 4456 generic.go:356] "Generic (PLEG): container finished" podID="b9c04022-9e9e-4704-986e-ad2ef2993224" containerID="1d5a6c49ec752b5f0027fec7f9cf110cc2d82950732d0624a8be84cb8a6e5624" exitCode=0 Oct 31 02:55:48.575368 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.575253 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75lnw" event={"ID":"b9c04022-9e9e-4704-986e-ad2ef2993224","Type":"ContainerDied","Data":"1d5a6c49ec752b5f0027fec7f9cf110cc2d82950732d0624a8be84cb8a6e5624"} Oct 31 02:55:48.634069 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.634009 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-tls\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.634069 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.634063 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.634283 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.634107 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcqwq\" (UniqueName: \"kubernetes.io/projected/ebaf5bff-0da3-496e-8461-914f5f374daf-kube-api-access-pcqwq\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.634283 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.634183 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.634283 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.634231 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebaf5bff-0da3-496e-8461-914f5f374daf-metrics-client-ca\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.634283 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.634265 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.636004 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.635138 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebaf5bff-0da3-496e-8461-914f5f374daf-tmp\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.636004 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.635154 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebaf5bff-0da3-496e-8461-914f5f374daf-tmp\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.636004 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.635307 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-grpc-tls\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.636004 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.635394 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.636004 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.635415 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebaf5bff-0da3-496e-8461-914f5f374daf-metrics-client-ca\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.637693 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.637663 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.637802 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.637725 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-tls\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.638075 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.638023 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.640884 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.638958 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.640884 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.639328 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-grpc-tls\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.641688 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.641663 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ebaf5bff-0da3-496e-8461-914f5f374daf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.647549 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.647527 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcqwq\" (UniqueName: \"kubernetes.io/projected/ebaf5bff-0da3-496e-8461-914f5f374daf-kube-api-access-pcqwq\") pod \"thanos-querier-678874f684-2zvzr\" (UID: \"ebaf5bff-0da3-496e-8461-914f5f374daf\") " pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:48.790169 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:48.790139 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:49.170396 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.170368 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-678874f684-2zvzr"] Oct 31 02:55:49.175690 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:49.175601 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebaf5bff_0da3_496e_8461_914f5f374daf.slice/crio-cdd5ebf0f2203ff9c325a4777871bf07da315fcf1c1badadd57978789629ffb0 WatchSource:0}: Error finding container cdd5ebf0f2203ff9c325a4777871bf07da315fcf1c1badadd57978789629ffb0: Status 404 returned error can't find the container with id cdd5ebf0f2203ff9c325a4777871bf07da315fcf1c1badadd57978789629ffb0 Oct 31 02:55:49.548729 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.548696 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kjmq9" Oct 31 02:55:49.586166 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.583135 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" event={"ID":"9e6cae46-c89a-458b-93a5-e313e7c01403","Type":"ContainerStarted","Data":"49007f7698bac9debe8bd6b04fbf541a49ca9dbb4ef14c2812fd6e84af35b7b8"} Oct 31 02:55:49.586166 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.583205 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" event={"ID":"9e6cae46-c89a-458b-93a5-e313e7c01403","Type":"ContainerStarted","Data":"fb9a65b7d6bf0c74c70e32cd956057dd261786d1728f082532bf7d1ccb47e169"} Oct 31 02:55:49.586166 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.583233 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" event={"ID":"9e6cae46-c89a-458b-93a5-e313e7c01403","Type":"ContainerStarted","Data":"86659c2ac67a08efe0b85f9138ff658b85d1cde83e0116700c94379cd6c9e68b"} Oct 31 02:55:49.586374 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.586257 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75lnw" event={"ID":"b9c04022-9e9e-4704-986e-ad2ef2993224","Type":"ContainerStarted","Data":"cf8e393ecec4befafc4e52a15af563f36c408ce4d031fe268966ecc19a4cdd28"} Oct 31 02:55:49.586374 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.586297 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75lnw" event={"ID":"b9c04022-9e9e-4704-986e-ad2ef2993224","Type":"ContainerStarted","Data":"d4b28fd1dfa0aafd4624989d4271b419f8ef876c7d2954db09bd3e5487a979de"} Oct 31 02:55:49.590372 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.589925 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" event={"ID":"12194edf-58dc-4167-a1e0-8888ccf43cc9","Type":"ContainerStarted","Data":"d1c0bb8d587444e28b87ac8fab07d3eddaa6160f983da617cb494836c48778a7"} Oct 31 02:55:49.591461 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.591417 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" event={"ID":"ebaf5bff-0da3-496e-8461-914f5f374daf","Type":"ContainerStarted","Data":"cdd5ebf0f2203ff9c325a4777871bf07da315fcf1c1badadd57978789629ffb0"} Oct 31 02:55:49.608054 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.607978 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hrd9c" podStartSLOduration=1.635073399 podStartE2EDuration="3.607962693s" totalImagesPullingTime="1.972889294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:49.606594768 +0000 UTC m=+31.287120897" watchObservedRunningTime="2026-10-31 02:55:49.607962693 +0000 UTC m=+31.288488822" Oct 31 02:55:49.633550 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.633508 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hzk4k" podStartSLOduration=1.73878831 podStartE2EDuration="3.633496201s" totalImagesPullingTime="1.894707891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:49.633156197 +0000 UTC m=+31.313682326" watchObservedRunningTime="2026-10-31 02:55:49.633496201 +0000 UTC m=+31.314022328" Oct 31 02:55:49.658587 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:49.658515 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-75lnw" podStartSLOduration=2.054367642 podStartE2EDuration="3.658498639s" totalImagesPullingTime="1.514999563s" totalInitContainerRuntime="89.131434ms" isStatefulPod=true podCreationTimestamp="2026-10-31 02:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:49.658016035 +0000 UTC m=+31.338542163" watchObservedRunningTime="2026-10-31 02:55:49.658498639 +0000 UTC m=+31.339024767" Oct 31 02:55:50.597581 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:50.597545 4456 generic.go:356] "Generic (PLEG): container finished" podID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerID="38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5" exitCode=0 Oct 31 02:55:50.598071 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:50.597614 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerDied","Data":"38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5"} Oct 31 02:55:51.046697 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.046657 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z"] Oct 31 02:55:51.060834 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.060805 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z"] Oct 31 02:55:51.060985 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.060924 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" Oct 31 02:55:51.067770 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.067746 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 02:55:51.068441 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.068417 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c56lj\"" Oct 31 02:55:51.156168 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.156133 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fzv6z\" (UID: \"0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" Oct 31 02:55:51.156362 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.156205 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q922h\" (UniqueName: \"kubernetes.io/projected/0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06-kube-api-access-q922h\") pod \"monitoring-plugin-77f4f76594-fzv6z\" (UID: \"0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" Oct 31 02:55:51.257172 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.257132 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fzv6z\" (UID: \"0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" Oct 31 02:55:51.257344 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.257210 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q922h\" (UniqueName: \"kubernetes.io/projected/0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06-kube-api-access-q922h\") pod \"monitoring-plugin-77f4f76594-fzv6z\" (UID: \"0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" Oct 31 02:55:51.269673 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.269643 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fzv6z\" (UID: \"0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" Oct 31 02:55:51.271135 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.271084 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q922h\" (UniqueName: \"kubernetes.io/projected/0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06-kube-api-access-q922h\") pod \"monitoring-plugin-77f4f76594-fzv6z\" (UID: \"0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" Oct 31 02:55:51.394462 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.394389 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" Oct 31 02:55:51.684623 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:51.684599 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z"] Oct 31 02:55:51.693411 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:51.693377 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3a2f93_5cb8_4c2b_8096_2c26b0bc4d06.slice/crio-1c37e8b8108a2384dcc684cb7d619e8cda8288acf7b35b91788497722de9e7fb WatchSource:0}: Error finding container 1c37e8b8108a2384dcc684cb7d619e8cda8288acf7b35b91788497722de9e7fb: Status 404 returned error can't find the container with id 1c37e8b8108a2384dcc684cb7d619e8cda8288acf7b35b91788497722de9e7fb Oct 31 02:55:52.609469 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:52.609431 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" event={"ID":"0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06","Type":"ContainerStarted","Data":"1c37e8b8108a2384dcc684cb7d619e8cda8288acf7b35b91788497722de9e7fb"} Oct 31 02:55:52.619565 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:52.617112 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" event={"ID":"ebaf5bff-0da3-496e-8461-914f5f374daf","Type":"ContainerStarted","Data":"16db98ea8550b1b0d11c34241a24cde55b02f8fb50c9acfdbeb200e38cdaace0"} Oct 31 02:55:52.619565 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:52.617184 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" event={"ID":"ebaf5bff-0da3-496e-8461-914f5f374daf","Type":"ContainerStarted","Data":"77cde7c18ca8d77220a1da9b9f8889c57d146d93070bb84238384e8469a92741"} Oct 31 02:55:52.619565 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:52.617213 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" event={"ID":"ebaf5bff-0da3-496e-8461-914f5f374daf","Type":"ContainerStarted","Data":"d0b46f3081a655aba7ca154e68d205cad5cc91f18fb5ab641d0eb6ae88529185"} Oct 31 02:55:53.625236 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.625201 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerStarted","Data":"2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e"} Oct 31 02:55:53.625618 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.625296 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerStarted","Data":"e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4"} Oct 31 02:55:53.625618 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.625334 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerStarted","Data":"bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3"} Oct 31 02:55:53.625618 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.625360 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerStarted","Data":"a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c"} Oct 31 02:55:53.625618 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.625384 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerStarted","Data":"0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee"} Oct 31 02:55:53.626861 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.626832 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" event={"ID":"0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06","Type":"ContainerStarted","Data":"d9e031028329b974650ca295ab761ad3feb99b428ec9346aa260f0932b88f301"} Oct 31 02:55:53.626945 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.626889 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" Oct 31 02:55:53.630508 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.630486 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" event={"ID":"ebaf5bff-0da3-496e-8461-914f5f374daf","Type":"ContainerStarted","Data":"198c5250876fcfc901cd3f9ce6e29446921129166346bb65b91860399a41def0"} Oct 31 02:55:53.630738 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.630556 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" event={"ID":"ebaf5bff-0da3-496e-8461-914f5f374daf","Type":"ContainerStarted","Data":"c9c4fd1c74b11ff0ad61968682289ce12413d3baf6210edf7f44a0bed5109543"} Oct 31 02:55:53.630738 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.630600 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" event={"ID":"ebaf5bff-0da3-496e-8461-914f5f374daf","Type":"ContainerStarted","Data":"07a521a01e362fbd2ede58eb01cad17b7f8c8249e5303eb30bd92d86f35a2f04"} Oct 31 02:55:53.630738 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.630648 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:53.632803 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.632786 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" Oct 31 02:55:53.644731 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.644707 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-kqjnj"] Oct 31 02:55:53.648082 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.648029 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kqjnj" Oct 31 02:55:53.651548 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.651510 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzv6z" podStartSLOduration=0.939227193 podStartE2EDuration="2.651499573s" totalImagesPullingTime="1.71227238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:53.650917259 +0000 UTC m=+35.331443387" watchObservedRunningTime="2026-10-31 02:55:53.651499573 +0000 UTC m=+35.332025700" Oct 31 02:55:53.654706 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.654686 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 02:55:53.659157 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.658648 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kqjnj"] Oct 31 02:55:53.659314 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.659301 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 02:55:53.684342 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.684267 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" podStartSLOduration=1.697572279 podStartE2EDuration="5.68425551s" totalImagesPullingTime="3.986683231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:53.68358901 +0000 UTC m=+35.364115145" watchObservedRunningTime="2026-10-31 02:55:53.68425551 +0000 UTC m=+35.364781688" Oct 31 02:55:53.782466 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.782435 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31079682-23c1-4203-b654-3a9c93b56901-tmp\") pod \"downloads-6dd789c8b7-kqjnj\" (UID: \"31079682-23c1-4203-b654-3a9c93b56901\") " pod="openshift-console/downloads-6dd789c8b7-kqjnj" Oct 31 02:55:53.782755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.782725 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh65z\" (UniqueName: \"kubernetes.io/projected/31079682-23c1-4203-b654-3a9c93b56901-kube-api-access-kh65z\") pod \"downloads-6dd789c8b7-kqjnj\" (UID: \"31079682-23c1-4203-b654-3a9c93b56901\") " pod="openshift-console/downloads-6dd789c8b7-kqjnj" Oct 31 02:55:53.883996 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.883964 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31079682-23c1-4203-b654-3a9c93b56901-tmp\") pod \"downloads-6dd789c8b7-kqjnj\" (UID: \"31079682-23c1-4203-b654-3a9c93b56901\") " pod="openshift-console/downloads-6dd789c8b7-kqjnj" Oct 31 02:55:53.884147 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.884013 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh65z\" (UniqueName: \"kubernetes.io/projected/31079682-23c1-4203-b654-3a9c93b56901-kube-api-access-kh65z\") pod \"downloads-6dd789c8b7-kqjnj\" (UID: \"31079682-23c1-4203-b654-3a9c93b56901\") " pod="openshift-console/downloads-6dd789c8b7-kqjnj" Oct 31 02:55:53.884363 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.884343 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31079682-23c1-4203-b654-3a9c93b56901-tmp\") pod \"downloads-6dd789c8b7-kqjnj\" (UID: \"31079682-23c1-4203-b654-3a9c93b56901\") " pod="openshift-console/downloads-6dd789c8b7-kqjnj" Oct 31 02:55:53.899339 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.899313 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh65z\" (UniqueName: \"kubernetes.io/projected/31079682-23c1-4203-b654-3a9c93b56901-kube-api-access-kh65z\") pod \"downloads-6dd789c8b7-kqjnj\" (UID: \"31079682-23c1-4203-b654-3a9c93b56901\") " pod="openshift-console/downloads-6dd789c8b7-kqjnj" Oct 31 02:55:53.962330 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:53.962239 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kqjnj" Oct 31 02:55:54.233356 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:54.233334 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kqjnj"] Oct 31 02:55:54.235905 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:54.235874 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31079682_23c1_4203_b654_3a9c93b56901.slice/crio-7b778e7e15ecbdf746ca744874623c6259217d9dcc949b8b9907fadd72883103 WatchSource:0}: Error finding container 7b778e7e15ecbdf746ca744874623c6259217d9dcc949b8b9907fadd72883103: Status 404 returned error can't find the container with id 7b778e7e15ecbdf746ca744874623c6259217d9dcc949b8b9907fadd72883103 Oct 31 02:55:54.641504 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:54.641417 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerStarted","Data":"89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699"} Oct 31 02:55:54.642738 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:54.642710 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kqjnj" event={"ID":"31079682-23c1-4203-b654-3a9c93b56901","Type":"ContainerStarted","Data":"7b778e7e15ecbdf746ca744874623c6259217d9dcc949b8b9907fadd72883103"} Oct 31 02:55:54.682017 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:54.681972 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.301136531 podStartE2EDuration="7.681960537s" totalImagesPullingTime="4.365048254s" totalInitContainerRuntime="15.775752ms" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:54.679099165 +0000 UTC m=+36.359625292" watchObservedRunningTime="2026-10-31 02:55:54.681960537 +0000 UTC m=+36.362486664" Oct 31 02:55:58.688711 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.688664 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-99cc849dd-kvzv9"] Oct 31 02:55:58.694999 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.694977 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.705775 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.705749 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 02:55:58.705902 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.705790 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 02:55:58.705902 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.705837 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 02:55:58.705902 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.705750 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 02:55:58.706081 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.706009 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vk5pn\"" Oct 31 02:55:58.706081 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.705797 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 02:55:58.707312 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.707291 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-99cc849dd-kvzv9"] Oct 31 02:55:58.830984 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.830949 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/230e3363-1397-4a61-92ef-3a29ae22ea13-tmp\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.830984 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.830992 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-serving-cert\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.831263 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.831055 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-oauth-serving-cert\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.831263 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.831196 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98h2l\" (UniqueName: \"kubernetes.io/projected/230e3363-1397-4a61-92ef-3a29ae22ea13-kube-api-access-98h2l\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.831263 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.831246 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-service-ca\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.831418 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.831285 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-console-config\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.831418 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.831314 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-oauth-config\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.932603 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.932567 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-oauth-serving-cert\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.932793 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.932614 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98h2l\" (UniqueName: \"kubernetes.io/projected/230e3363-1397-4a61-92ef-3a29ae22ea13-kube-api-access-98h2l\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.932793 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.932647 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-service-ca\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.932793 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.932673 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-console-config\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.932793 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.932696 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-oauth-config\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.932793 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.932739 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/230e3363-1397-4a61-92ef-3a29ae22ea13-tmp\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.932793 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.932767 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-serving-cert\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.933470 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.933429 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-oauth-serving-cert\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.934162 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.933595 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/230e3363-1397-4a61-92ef-3a29ae22ea13-tmp\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.934162 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.933892 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-service-ca\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.934162 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.934114 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-console-config\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.935685 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.935663 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-oauth-config\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.935820 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.935800 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-serving-cert\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:58.978239 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:58.978177 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98h2l\" (UniqueName: \"kubernetes.io/projected/230e3363-1397-4a61-92ef-3a29ae22ea13-kube-api-access-98h2l\") pod \"console-99cc849dd-kvzv9\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:59.011987 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:59.011949 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:55:59.300393 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:59.300298 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-99cc849dd-kvzv9"] Oct 31 02:55:59.304796 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:55:59.304765 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230e3363_1397_4a61_92ef_3a29ae22ea13.slice/crio-2681ce5dfd98415a7a609b37bf979235fd4bdae1d27cd5dacfc501fa392b105a WatchSource:0}: Error finding container 2681ce5dfd98415a7a609b37bf979235fd4bdae1d27cd5dacfc501fa392b105a: Status 404 returned error can't find the container with id 2681ce5dfd98415a7a609b37bf979235fd4bdae1d27cd5dacfc501fa392b105a Oct 31 02:55:59.650337 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:59.650254 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-678874f684-2zvzr" Oct 31 02:55:59.666398 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:55:59.666364 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99cc849dd-kvzv9" event={"ID":"230e3363-1397-4a61-92ef-3a29ae22ea13","Type":"ContainerStarted","Data":"2681ce5dfd98415a7a609b37bf979235fd4bdae1d27cd5dacfc501fa392b105a"} Oct 31 02:56:02.137519 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:56:02.137432 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 02:56:02.682007 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:02.681954 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99cc849dd-kvzv9" event={"ID":"230e3363-1397-4a61-92ef-3a29ae22ea13","Type":"ContainerStarted","Data":"3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b"} Oct 31 02:56:02.711502 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:02.711443 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-99cc849dd-kvzv9" podStartSLOduration=1.478169858 podStartE2EDuration="4.711426768s" totalImagesPullingTime="3.23325691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:56:02.710399069 +0000 UTC m=+44.390925195" watchObservedRunningTime="2026-10-31 02:56:02.711426768 +0000 UTC m=+44.391952891" Oct 31 02:56:09.012210 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:09.012164 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:56:09.012701 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:09.012236 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:56:09.017388 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:09.017364 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:56:09.725601 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:09.725561 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:56:10.109363 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.109289 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c49b66d86-zjs5z"] Oct 31 02:56:10.114405 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.114380 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.127158 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.127117 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 02:56:10.128572 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.128546 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c49b66d86-zjs5z"] Oct 31 02:56:10.137742 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.137699 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfghh\" (UniqueName: \"kubernetes.io/projected/6e12e21d-4512-4211-b710-b6b6048622d2-kube-api-access-wfghh\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.137742 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.137738 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-trusted-ca-bundle\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.137941 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.137865 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-console-config\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.137941 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.137907 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e12e21d-4512-4211-b710-b6b6048622d2-tmp\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.138076 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.137938 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-serving-cert\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.138076 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.137970 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-oauth-serving-cert\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.138191 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.138093 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-oauth-config\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.138191 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.138136 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-service-ca\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.239079 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.239015 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-serving-cert\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.239079 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.239082 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-oauth-serving-cert\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.239326 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.239137 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-oauth-config\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.239326 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.239173 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-service-ca\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.239326 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.239217 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfghh\" (UniqueName: \"kubernetes.io/projected/6e12e21d-4512-4211-b710-b6b6048622d2-kube-api-access-wfghh\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.239326 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.239249 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-trusted-ca-bundle\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.239326 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.239286 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-console-config\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.239326 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.239307 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e12e21d-4512-4211-b710-b6b6048622d2-tmp\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.239703 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.239680 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e12e21d-4512-4211-b710-b6b6048622d2-tmp\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.240071 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.240029 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-oauth-serving-cert\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.240176 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.240155 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-service-ca\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.240464 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.240440 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-console-config\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.240538 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.240517 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-trusted-ca-bundle\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.241849 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.241829 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-oauth-config\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.242018 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.241996 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-serving-cert\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.254278 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.254257 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfghh\" (UniqueName: \"kubernetes.io/projected/6e12e21d-4512-4211-b710-b6b6048622d2-kube-api-access-wfghh\") pod \"console-5c49b66d86-zjs5z\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.429751 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.429718 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:10.708211 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.708182 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c49b66d86-zjs5z"] Oct 31 02:56:10.710156 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:56:10.710119 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e12e21d_4512_4211_b710_b6b6048622d2.slice/crio-b9cffbf23d55987d7d7ca0bfd18b3b85f7d525fd640ee6520f35b80535cdb01c WatchSource:0}: Error finding container b9cffbf23d55987d7d7ca0bfd18b3b85f7d525fd640ee6520f35b80535cdb01c: Status 404 returned error can't find the container with id b9cffbf23d55987d7d7ca0bfd18b3b85f7d525fd640ee6520f35b80535cdb01c Oct 31 02:56:10.726874 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.726845 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kqjnj" event={"ID":"31079682-23c1-4203-b654-3a9c93b56901","Type":"ContainerStarted","Data":"deee4217ef08acbdadeffcdd0df571d657fe7f3637d1f355ec0eb210f53a692f"} Oct 31 02:56:10.727125 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.727097 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-kqjnj" Oct 31 02:56:10.728394 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.728368 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c49b66d86-zjs5z" event={"ID":"6e12e21d-4512-4211-b710-b6b6048622d2","Type":"ContainerStarted","Data":"b9cffbf23d55987d7d7ca0bfd18b3b85f7d525fd640ee6520f35b80535cdb01c"} Oct 31 02:56:10.735249 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.735142 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-kqjnj" Oct 31 02:56:10.749331 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:10.748981 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-kqjnj" podStartSLOduration=1.50920768 podStartE2EDuration="17.74896374s" totalImagesPullingTime="16.23975606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:56:10.747098786 +0000 UTC m=+52.427624914" watchObservedRunningTime="2026-10-31 02:56:10.74896374 +0000 UTC m=+52.429489868" Oct 31 02:56:11.735017 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:11.734980 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c49b66d86-zjs5z" event={"ID":"6e12e21d-4512-4211-b710-b6b6048622d2","Type":"ContainerStarted","Data":"c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309"} Oct 31 02:56:11.758770 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:11.758720 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c49b66d86-zjs5z" podStartSLOduration=1.758706007 podStartE2EDuration="1.758706007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:56:11.75788064 +0000 UTC m=+53.438406768" watchObservedRunningTime="2026-10-31 02:56:11.758706007 +0000 UTC m=+53.439232138" Oct 31 02:56:20.433257 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:20.433228 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:20.433715 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:20.433283 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:20.434908 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:20.434886 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:20.780175 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:20.780110 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:56:20.837320 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:20.837288 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-99cc849dd-kvzv9"] Oct 31 02:56:32.132680 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:56:32.132642 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 02:56:36.190826 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.190797 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 02:56:36.191282 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.191112 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="alertmanager" containerID="cri-o://0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee" gracePeriod=120 Oct 31 02:56:36.191282 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.191152 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy-web" containerID="cri-o://bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3" gracePeriod=120 Oct 31 02:56:36.191282 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.191169 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy" containerID="cri-o://e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4" gracePeriod=120 Oct 31 02:56:36.191282 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.191193 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy-metric" containerID="cri-o://2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e" gracePeriod=120 Oct 31 02:56:36.191282 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.191221 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="config-reloader" containerID="cri-o://a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c" gracePeriod=120 Oct 31 02:56:36.191282 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.191176 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="prom-label-proxy" containerID="cri-o://89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699" gracePeriod=120 Oct 31 02:56:36.836324 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.836289 4456 generic.go:356] "Generic (PLEG): container finished" podID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerID="89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699" exitCode=0 Oct 31 02:56:36.836324 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.836318 4456 generic.go:356] "Generic (PLEG): container finished" podID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerID="2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e" exitCode=0 Oct 31 02:56:36.836324 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.836325 4456 generic.go:356] "Generic (PLEG): container finished" podID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerID="e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4" exitCode=0 Oct 31 02:56:36.836324 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.836332 4456 generic.go:356] "Generic (PLEG): container finished" podID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerID="a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c" exitCode=0 Oct 31 02:56:36.836579 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.836338 4456 generic.go:356] "Generic (PLEG): container finished" podID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerID="0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee" exitCode=0 Oct 31 02:56:36.836579 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.836357 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerDied","Data":"89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699"} Oct 31 02:56:36.836579 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.836417 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerDied","Data":"2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e"} Oct 31 02:56:36.836579 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.836442 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerDied","Data":"e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4"} Oct 31 02:56:36.836579 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.836466 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerDied","Data":"a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c"} Oct 31 02:56:36.836579 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:36.836491 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerDied","Data":"0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee"} Oct 31 02:56:37.455209 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.455185 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:37.579750 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.579684 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-tls-assets\" (UniqueName: \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-tls-assets\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.579750 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.579723 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.579973 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.579754 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-kube-api-access-nsttz\" (UniqueName: \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-kube-api-access-nsttz\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.579973 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.579899 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-main-db\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.579973 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.579959 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-web-config\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-web-config\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.580165 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.580001 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-config-volume\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-config-volume\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.580165 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.580070 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-config-out\" (UniqueName: \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-config-out\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.580165 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.580099 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-trusted-ca-bundle\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.580165 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.580131 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-cluster-tls-config\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.580387 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.580179 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-web\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.580387 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.580299 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-metrics-client-ca\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.580387 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.580314 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-main-db" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:56:37.580387 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.580340 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-main-tls\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.580387 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.580372 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-metric\") pod \"777b7650-dacc-43f8-ad77-a9ae3a593410\" (UID: \"777b7650-dacc-43f8-ad77-a9ae3a593410\") " Oct 31 02:56:37.581116 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.580648 4456 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-main-db\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.581116 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.581049 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-metrics-client-ca" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:37.581418 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.581388 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-trusted-ca-bundle" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:37.582567 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.582539 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-tls-assets" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:37.583344 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.583303 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-config-volume" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:37.584197 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.584082 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-kube-api-access-nsttz" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "kube-api-access-nsttz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:37.584197 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.584170 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-metric" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:37.584453 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.584421 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-web" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:37.585012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.584985 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:37.585930 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.585893 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-config-out" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:56:37.586020 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.585953 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-main-tls" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:37.590264 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.590224 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-cluster-tls-config" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:37.598000 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.597976 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-web-config" pod "777b7650-dacc-43f8-ad77-a9ae3a593410" (UID: "777b7650-dacc-43f8-ad77-a9ae3a593410"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:37.681889 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681861 4456 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-main-tls\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.681889 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681887 4456 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.682101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681900 4456 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-tls-assets\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.682101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681909 4456 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.682101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681918 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsttz\" (UniqueName: \"kubernetes.io/projected/777b7650-dacc-43f8-ad77-a9ae3a593410-kube-api-access-nsttz\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.682101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681927 4456 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-web-config\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.682101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681934 4456 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.682101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681942 4456 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/777b7650-dacc-43f8-ad77-a9ae3a593410-config-out\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.682101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681951 4456 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.682101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681959 4456 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-cluster-tls-config\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.682101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681967 4456 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/777b7650-dacc-43f8-ad77-a9ae3a593410-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.682101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.681976 4456 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/777b7650-dacc-43f8-ad77-a9ae3a593410-metrics-client-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:37.842818 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.842745 4456 generic.go:356] "Generic (PLEG): container finished" podID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerID="bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3" exitCode=0 Oct 31 02:56:37.842818 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.842806 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerDied","Data":"bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3"} Oct 31 02:56:37.843028 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.842833 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:37.843028 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.842861 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"777b7650-dacc-43f8-ad77-a9ae3a593410","Type":"ContainerDied","Data":"04285c4f1bb7c636397e53af87e400736103bec67cf339cfd9d1e86546dc0614"} Oct 31 02:56:37.843028 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.842892 4456 scope.go:122] "RemoveContainer" containerID="89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699" Oct 31 02:56:37.851272 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.851251 4456 scope.go:122] "RemoveContainer" containerID="2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e" Oct 31 02:56:37.857972 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.857952 4456 scope.go:122] "RemoveContainer" containerID="e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4" Oct 31 02:56:37.864256 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.864238 4456 scope.go:122] "RemoveContainer" containerID="bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3" Oct 31 02:56:37.870408 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.870389 4456 scope.go:122] "RemoveContainer" containerID="a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c" Oct 31 02:56:37.876900 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.876881 4456 scope.go:122] "RemoveContainer" containerID="0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee" Oct 31 02:56:37.883674 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.883659 4456 scope.go:122] "RemoveContainer" containerID="38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5" Oct 31 02:56:37.890337 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.890320 4456 scope.go:122] "RemoveContainer" containerID="89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699" Oct 31 02:56:37.890613 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:56:37.890593 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699\": container with ID starting with 89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699 not found: ID does not exist" containerID="89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699" Oct 31 02:56:37.890704 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.890618 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699"} err="failed to get container status \"89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699\": rpc error: code = NotFound desc = could not find container \"89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699\": container with ID starting with 89ecda7f4612a6b7d857b997afdad59154f431964e64d87454fcc45574173699 not found: ID does not exist" Oct 31 02:56:37.890704 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.890648 4456 scope.go:122] "RemoveContainer" containerID="2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e" Oct 31 02:56:37.890909 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:56:37.890886 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e\": container with ID starting with 2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e not found: ID does not exist" containerID="2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e" Oct 31 02:56:37.890958 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.890912 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e"} err="failed to get container status \"2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e\": rpc error: code = NotFound desc = could not find container \"2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e\": container with ID starting with 2af42d7bebb56f7049e08604a06efce17054558bee10d7e0743da73614046e6e not found: ID does not exist" Oct 31 02:56:37.890958 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.890925 4456 scope.go:122] "RemoveContainer" containerID="e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4" Oct 31 02:56:37.891214 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:56:37.891189 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4\": container with ID starting with e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4 not found: ID does not exist" containerID="e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4" Oct 31 02:56:37.891276 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.891226 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4"} err="failed to get container status \"e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4\": rpc error: code = NotFound desc = could not find container \"e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4\": container with ID starting with e9a9d27304ad0b3d37777267e255d91c65ab95d0a8ee2abc6ac08c4dc466f6e4 not found: ID does not exist" Oct 31 02:56:37.891276 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.891246 4456 scope.go:122] "RemoveContainer" containerID="bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3" Oct 31 02:56:37.891549 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:56:37.891531 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3\": container with ID starting with bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3 not found: ID does not exist" containerID="bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3" Oct 31 02:56:37.891603 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.891553 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3"} err="failed to get container status \"bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3\": rpc error: code = NotFound desc = could not find container \"bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3\": container with ID starting with bdd76cc64d598ea82a29316c02d395c9b19ef5ab2415eea9c29c20796d9126c3 not found: ID does not exist" Oct 31 02:56:37.891603 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.891574 4456 scope.go:122] "RemoveContainer" containerID="a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c" Oct 31 02:56:37.891828 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:56:37.891807 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c\": container with ID starting with a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c not found: ID does not exist" containerID="a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c" Oct 31 02:56:37.891932 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.891832 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c"} err="failed to get container status \"a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c\": rpc error: code = NotFound desc = could not find container \"a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c\": container with ID starting with a405ec02cd622b0c889faa4d2741c58b8c83878200b302365cf160b742b4f66c not found: ID does not exist" Oct 31 02:56:37.891932 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.891855 4456 scope.go:122] "RemoveContainer" containerID="0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee" Oct 31 02:56:37.892220 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:56:37.892199 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee\": container with ID starting with 0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee not found: ID does not exist" containerID="0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee" Oct 31 02:56:37.892323 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.892226 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee"} err="failed to get container status \"0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee\": rpc error: code = NotFound desc = could not find container \"0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee\": container with ID starting with 0fd74acb0c69e852ade0018ddd00363dfce515c912290a79538981b43a336bee not found: ID does not exist" Oct 31 02:56:37.892323 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.892246 4456 scope.go:122] "RemoveContainer" containerID="38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5" Oct 31 02:56:37.892534 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:56:37.892516 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5\": container with ID starting with 38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5 not found: ID does not exist" containerID="38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5" Oct 31 02:56:37.892599 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.892550 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5"} err="failed to get container status \"38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5\": rpc error: code = NotFound desc = could not find container \"38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5\": container with ID starting with 38c7122478ddbc5de932ce164ddf748cbf3a863cdb1f80f6b6b3b9f78d5975e5 not found: ID does not exist" Oct 31 02:56:37.925261 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.925230 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 02:56:37.935748 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.935699 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 02:56:37.946080 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946058 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 02:56:37.946396 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946379 4456 cpu_manager.go:414] "containerMap: removing container" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="init-config-reloader" Oct 31 02:56:37.946473 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946399 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="init-config-reloader" Oct 31 02:56:37.946473 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946425 4456 cpu_manager.go:414] "containerMap: removing container" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="config-reloader" Oct 31 02:56:37.946473 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946434 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="config-reloader" Oct 31 02:56:37.946473 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946449 4456 cpu_manager.go:414] "containerMap: removing container" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="alertmanager" Oct 31 02:56:37.946473 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946456 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="alertmanager" Oct 31 02:56:37.946473 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946466 4456 cpu_manager.go:414] "containerMap: removing container" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy-metric" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946474 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy-metric" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946493 4456 cpu_manager.go:414] "containerMap: removing container" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="prom-label-proxy" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946501 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="prom-label-proxy" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946514 4456 cpu_manager.go:414] "containerMap: removing container" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946521 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946538 4456 cpu_manager.go:414] "containerMap: removing container" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy-web" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946545 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy-web" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946648 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="prom-label-proxy" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946662 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946673 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy-web" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946686 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="config-reloader" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946696 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="alertmanager" Oct 31 02:56:37.946755 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.946703 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" containerName="kube-rbac-proxy-metric" Oct 31 02:56:37.983365 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.983339 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 02:56:37.983498 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.983467 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:37.991452 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.991281 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 02:56:37.991452 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.991316 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 02:56:37.991452 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.991356 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 02:56:37.991452 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.991295 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 02:56:37.991452 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.991356 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 02:56:37.991452 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.991356 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 02:56:37.991452 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.991400 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 02:56:37.991452 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.991402 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 02:56:37.991452 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:37.991408 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wv4jz\"" Oct 31 02:56:38.011511 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.011492 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 02:56:38.085135 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085107 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085135 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085135 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085308 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085158 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94846398-db50-47bb-8382-30438dc3bf34-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085308 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085211 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94846398-db50-47bb-8382-30438dc3bf34-config-out\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085308 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085278 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2llwp\" (UniqueName: \"kubernetes.io/projected/94846398-db50-47bb-8382-30438dc3bf34-kube-api-access-2llwp\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085430 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085315 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085430 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085346 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94846398-db50-47bb-8382-30438dc3bf34-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085430 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085373 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085430 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085394 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-web-config\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085566 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085434 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94846398-db50-47bb-8382-30438dc3bf34-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085566 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085490 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-config-volume\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085566 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085525 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.085566 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.085544 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94846398-db50-47bb-8382-30438dc3bf34-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.185844 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.185819 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94846398-db50-47bb-8382-30438dc3bf34-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186001 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.185851 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186001 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.185876 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-web-config\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186001 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.185892 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94846398-db50-47bb-8382-30438dc3bf34-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186001 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.185910 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-config-volume\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186001 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.185951 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186001 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.185974 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94846398-db50-47bb-8382-30438dc3bf34-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186001 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.186000 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186363 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.186026 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186363 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.186087 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94846398-db50-47bb-8382-30438dc3bf34-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186363 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.186119 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94846398-db50-47bb-8382-30438dc3bf34-config-out\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186363 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.186157 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2llwp\" (UniqueName: \"kubernetes.io/projected/94846398-db50-47bb-8382-30438dc3bf34-kube-api-access-2llwp\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.186363 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.186182 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.187308 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.186749 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94846398-db50-47bb-8382-30438dc3bf34-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.187308 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.186898 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94846398-db50-47bb-8382-30438dc3bf34-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.187308 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.186936 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94846398-db50-47bb-8382-30438dc3bf34-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.188859 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.188836 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-web-config\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.188933 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.188890 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94846398-db50-47bb-8382-30438dc3bf34-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.188993 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.188972 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.189520 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.189436 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94846398-db50-47bb-8382-30438dc3bf34-config-out\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.189698 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.189680 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.189751 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.189727 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.189870 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.189851 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.189906 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.189851 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-config-volume\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.190737 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.190721 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94846398-db50-47bb-8382-30438dc3bf34-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.199982 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.199960 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2llwp\" (UniqueName: \"kubernetes.io/projected/94846398-db50-47bb-8382-30438dc3bf34-kube-api-access-2llwp\") pod \"alertmanager-main-0\" (UID: \"94846398-db50-47bb-8382-30438dc3bf34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.410160 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.410126 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 02:56:38.433461 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.433432 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777b7650-dacc-43f8-ad77-a9ae3a593410" path="/var/lib/kubelet/pods/777b7650-dacc-43f8-ad77-a9ae3a593410/volumes" Oct 31 02:56:38.714376 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.714350 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 02:56:38.714715 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:56:38.714527 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94846398_db50_47bb_8382_30438dc3bf34.slice/crio-c682b20362699d1547d0ebe474d60c69501dbca6509bcf78384610a9447d4aa2 WatchSource:0}: Error finding container c682b20362699d1547d0ebe474d60c69501dbca6509bcf78384610a9447d4aa2: Status 404 returned error can't find the container with id c682b20362699d1547d0ebe474d60c69501dbca6509bcf78384610a9447d4aa2 Oct 31 02:56:38.847836 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.847803 4456 generic.go:356] "Generic (PLEG): container finished" podID="94846398-db50-47bb-8382-30438dc3bf34" containerID="7ed6a55ece400a1ff458290929f6f1a93df0518b8b7dcc6d880018232b353438" exitCode=0 Oct 31 02:56:38.847948 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.847884 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94846398-db50-47bb-8382-30438dc3bf34","Type":"ContainerDied","Data":"7ed6a55ece400a1ff458290929f6f1a93df0518b8b7dcc6d880018232b353438"} Oct 31 02:56:38.848001 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:38.847961 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94846398-db50-47bb-8382-30438dc3bf34","Type":"ContainerStarted","Data":"c682b20362699d1547d0ebe474d60c69501dbca6509bcf78384610a9447d4aa2"} Oct 31 02:56:39.855370 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:39.855333 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94846398-db50-47bb-8382-30438dc3bf34","Type":"ContainerStarted","Data":"8be9f44dd6a0fee549dd802039c0429612d9e43266c4077d92b52d040bf79f57"} Oct 31 02:56:39.855763 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:39.855427 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94846398-db50-47bb-8382-30438dc3bf34","Type":"ContainerStarted","Data":"e248788a7e2a8067a4838c05a2a63161812f73e76a2dc1aad84a57415d3d8159"} Oct 31 02:56:39.855763 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:39.855453 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94846398-db50-47bb-8382-30438dc3bf34","Type":"ContainerStarted","Data":"42e93e8f4abe70b7a05d101cc99c8056d6a87e39c51ebe2964de3346b2e61b53"} Oct 31 02:56:39.855763 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:39.855478 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94846398-db50-47bb-8382-30438dc3bf34","Type":"ContainerStarted","Data":"3dee5312cc3c19e1aad20f6e13f4c5571f83444cb94b626c128f339f95cc88e4"} Oct 31 02:56:39.855763 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:39.855501 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94846398-db50-47bb-8382-30438dc3bf34","Type":"ContainerStarted","Data":"c3eccc837cc9b24a06fc3c4120c3c69d3b932be41e37abfdd30de2e6954d69ab"} Oct 31 02:56:39.855763 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:39.855522 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94846398-db50-47bb-8382-30438dc3bf34","Type":"ContainerStarted","Data":"d179e70a4dd4a9b8485d8e7ebb46520c7dbc72fe4d84eb4259c1d5500f7c7c91"} Oct 31 02:56:40.248471 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.248421 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.235679422 podStartE2EDuration="3.248408464s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.729042ms" isStatefulPod=false podCreationTimestamp="2026-10-31 02:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:56:40.221473078 +0000 UTC m=+81.901999205" watchObservedRunningTime="2026-10-31 02:56:40.248408464 +0000 UTC m=+81.928934590" Oct 31 02:56:40.457214 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.457182 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f56555f87-62h7x"] Oct 31 02:56:40.478881 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.478850 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.610733 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.610654 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2177bb5e-812d-4873-830a-2ab7e339c610-metrics-client-ca\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.610733 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.610696 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-telemeter-client-tls\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.610733 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.610719 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-federate-client-tls\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.610963 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.610780 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.610963 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.610910 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-secret-telemeter-client\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.611065 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.610983 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2177bb5e-812d-4873-830a-2ab7e339c610-serving-certs-ca-bundle\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.611065 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.611001 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29f6t\" (UniqueName: \"kubernetes.io/projected/2177bb5e-812d-4873-830a-2ab7e339c610-kube-api-access-29f6t\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.611065 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.611053 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2177bb5e-812d-4873-830a-2ab7e339c610-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.711987 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.711953 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2177bb5e-812d-4873-830a-2ab7e339c610-serving-certs-ca-bundle\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.711987 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.711986 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29f6t\" (UniqueName: \"kubernetes.io/projected/2177bb5e-812d-4873-830a-2ab7e339c610-kube-api-access-29f6t\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.712250 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.712014 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2177bb5e-812d-4873-830a-2ab7e339c610-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.712250 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.712071 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2177bb5e-812d-4873-830a-2ab7e339c610-metrics-client-ca\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.712250 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.712093 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-telemeter-client-tls\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.712250 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.712114 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-federate-client-tls\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.712250 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.712146 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.712250 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.712175 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-secret-telemeter-client\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.712812 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.712786 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2177bb5e-812d-4873-830a-2ab7e339c610-metrics-client-ca\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.759454 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.759422 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f56555f87-62h7x"] Oct 31 02:56:40.819355 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.819322 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 02:56:40.825565 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.825537 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.833354 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.833331 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4bx8q\"" Oct 31 02:56:40.852595 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.852564 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 02:56:40.852716 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.852629 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 02:56:40.857321 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.857298 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 02:56:40.857321 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.857315 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 02:56:40.863871 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.863815 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2177bb5e-812d-4873-830a-2ab7e339c610-serving-certs-ca-bundle\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.866260 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.866234 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-federate-client-tls\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.866533 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.866512 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-telemeter-client-tls\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:40.866709 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:40.866689 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2177bb5e-812d-4873-830a-2ab7e339c610-secret-telemeter-client\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:41.482062 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:41.482015 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 02:56:41.483699 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:41.483669 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2177bb5e-812d-4873-830a-2ab7e339c610-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:41.494944 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:41.494916 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29f6t\" (UniqueName: \"kubernetes.io/projected/2177bb5e-812d-4873-830a-2ab7e339c610-kube-api-access-29f6t\") pod \"telemeter-client-7f56555f87-62h7x\" (UID: \"2177bb5e-812d-4873-830a-2ab7e339c610\") " pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:41.760219 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:41.760142 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" Oct 31 02:56:42.073730 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:56:42.073698 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2177bb5e_812d_4873_830a_2ab7e339c610.slice/crio-2797b86a431718c2c763deeec322b49bcf585d3274f31cb065db955745816a4c WatchSource:0}: Error finding container 2797b86a431718c2c763deeec322b49bcf585d3274f31cb065db955745816a4c: Status 404 returned error can't find the container with id 2797b86a431718c2c763deeec322b49bcf585d3274f31cb065db955745816a4c Oct 31 02:56:42.079937 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:42.079908 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f56555f87-62h7x"] Oct 31 02:56:42.874106 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:42.874062 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" event={"ID":"2177bb5e-812d-4873-830a-2ab7e339c610","Type":"ContainerStarted","Data":"2797b86a431718c2c763deeec322b49bcf585d3274f31cb065db955745816a4c"} Oct 31 02:56:44.884259 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:44.884181 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" event={"ID":"2177bb5e-812d-4873-830a-2ab7e339c610","Type":"ContainerStarted","Data":"a702db22acc7ae72b1955f755aafd74d25a5cdfd5bf72c527c55cdae492fb25f"} Oct 31 02:56:44.884259 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:44.884236 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" event={"ID":"2177bb5e-812d-4873-830a-2ab7e339c610","Type":"ContainerStarted","Data":"08f243460e49f797b3f5ddfc272018ebb77840e563b093d0f62a350f6192b151"} Oct 31 02:56:44.884259 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:44.884254 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" event={"ID":"2177bb5e-812d-4873-830a-2ab7e339c610","Type":"ContainerStarted","Data":"f6ffa7c32f453886b45fe82beef929d416dfbb026df385dee7d5f1b2fda32b5f"} Oct 31 02:56:44.961467 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:44.961416 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f56555f87-62h7x" podStartSLOduration=2.797652571 podStartE2EDuration="4.961403747s" totalImagesPullingTime="2.163751176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:56:44.927500484 +0000 UTC m=+86.608026611" watchObservedRunningTime="2026-10-31 02:56:44.961403747 +0000 UTC m=+86.641929873" Oct 31 02:56:45.002531 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.002504 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84d474bb7b-vpljv"] Oct 31 02:56:45.024977 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.024944 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d474bb7b-vpljv"] Oct 31 02:56:45.025124 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.025063 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.152487 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.152455 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-oauth-serving-cert\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.152657 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.152493 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-serving-cert\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.152657 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.152603 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-service-ca\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.152657 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.152637 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-trusted-ca-bundle\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.152657 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.152658 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c6b015-ac0d-4729-af0b-258c96e6630c-tmp\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.152826 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.152756 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5949\" (UniqueName: \"kubernetes.io/projected/57c6b015-ac0d-4729-af0b-258c96e6630c-kube-api-access-p5949\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.152826 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.152789 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-oauth-config\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.152826 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.152805 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-console-config\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.253527 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.253500 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c6b015-ac0d-4729-af0b-258c96e6630c-tmp\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.253691 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.253541 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5949\" (UniqueName: \"kubernetes.io/projected/57c6b015-ac0d-4729-af0b-258c96e6630c-kube-api-access-p5949\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.253691 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.253563 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-oauth-config\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.253691 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.253578 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-console-config\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.253691 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.253613 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-oauth-serving-cert\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.253691 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.253631 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-serving-cert\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.253691 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.253665 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-service-ca\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.253691 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.253687 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-trusted-ca-bundle\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.254073 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.253959 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c6b015-ac0d-4729-af0b-258c96e6630c-tmp\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.254490 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.254464 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-console-config\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.254490 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.254476 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-oauth-serving-cert\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.254656 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.254498 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-service-ca\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.254656 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.254645 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-trusted-ca-bundle\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.256146 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.256127 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-oauth-config\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.256285 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.256268 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-serving-cert\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.296876 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.296848 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5949\" (UniqueName: \"kubernetes.io/projected/57c6b015-ac0d-4729-af0b-258c96e6630c-kube-api-access-p5949\") pod \"console-84d474bb7b-vpljv\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.348319 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.348289 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:45.654225 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:56:45.654139 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c6b015_ac0d_4729_af0b_258c96e6630c.slice/crio-25827de4f8d53392c001c03567256266f3256d41f125c8af865a56e67b1e2a02 WatchSource:0}: Error finding container 25827de4f8d53392c001c03567256266f3256d41f125c8af865a56e67b1e2a02: Status 404 returned error can't find the container with id 25827de4f8d53392c001c03567256266f3256d41f125c8af865a56e67b1e2a02 Oct 31 02:56:45.659868 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.659847 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d474bb7b-vpljv"] Oct 31 02:56:45.856804 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.856727 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-99cc849dd-kvzv9" podUID="230e3363-1397-4a61-92ef-3a29ae22ea13" containerName="console" containerID="cri-o://3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b" gracePeriod=15 Oct 31 02:56:45.892314 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.892105 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d474bb7b-vpljv" event={"ID":"57c6b015-ac0d-4729-af0b-258c96e6630c","Type":"ContainerStarted","Data":"e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812"} Oct 31 02:56:45.892314 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.892157 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d474bb7b-vpljv" event={"ID":"57c6b015-ac0d-4729-af0b-258c96e6630c","Type":"ContainerStarted","Data":"25827de4f8d53392c001c03567256266f3256d41f125c8af865a56e67b1e2a02"} Oct 31 02:56:45.953363 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:45.953260 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84d474bb7b-vpljv" podStartSLOduration=1.953240836 podStartE2EDuration="1.953240836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:56:45.949300089 +0000 UTC m=+87.629826216" watchObservedRunningTime="2026-10-31 02:56:45.953240836 +0000 UTC m=+87.633766963" Oct 31 02:56:46.125220 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.125194 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:56:46.263973 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.263949 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230e3363-1397-4a61-92ef-3a29ae22ea13-tmp\" (UniqueName: \"kubernetes.io/empty-dir/230e3363-1397-4a61-92ef-3a29ae22ea13-tmp\") pod \"230e3363-1397-4a61-92ef-3a29ae22ea13\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " Oct 31 02:56:46.264137 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264064 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-service-ca\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-service-ca\") pod \"230e3363-1397-4a61-92ef-3a29ae22ea13\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " Oct 31 02:56:46.264137 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264095 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-serving-cert\") pod \"230e3363-1397-4a61-92ef-3a29ae22ea13\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " Oct 31 02:56:46.264137 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264114 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-oauth-config\") pod \"230e3363-1397-4a61-92ef-3a29ae22ea13\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " Oct 31 02:56:46.264308 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264137 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-oauth-serving-cert\") pod \"230e3363-1397-4a61-92ef-3a29ae22ea13\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " Oct 31 02:56:46.264308 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264183 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230e3363-1397-4a61-92ef-3a29ae22ea13-kube-api-access-98h2l\" (UniqueName: \"kubernetes.io/projected/230e3363-1397-4a61-92ef-3a29ae22ea13-kube-api-access-98h2l\") pod \"230e3363-1397-4a61-92ef-3a29ae22ea13\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " Oct 31 02:56:46.264308 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264235 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-console-config\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-console-config\") pod \"230e3363-1397-4a61-92ef-3a29ae22ea13\" (UID: \"230e3363-1397-4a61-92ef-3a29ae22ea13\") " Oct 31 02:56:46.264460 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264230 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230e3363-1397-4a61-92ef-3a29ae22ea13-tmp" pod "230e3363-1397-4a61-92ef-3a29ae22ea13" (UID: "230e3363-1397-4a61-92ef-3a29ae22ea13"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:56:46.264516 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264501 4456 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/230e3363-1397-4a61-92ef-3a29ae22ea13-tmp\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:46.264657 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264634 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-service-ca" pod "230e3363-1397-4a61-92ef-3a29ae22ea13" (UID: "230e3363-1397-4a61-92ef-3a29ae22ea13"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:46.264813 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264788 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-oauth-serving-cert" pod "230e3363-1397-4a61-92ef-3a29ae22ea13" (UID: "230e3363-1397-4a61-92ef-3a29ae22ea13"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:46.264962 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.264871 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-console-config" pod "230e3363-1397-4a61-92ef-3a29ae22ea13" (UID: "230e3363-1397-4a61-92ef-3a29ae22ea13"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:46.266568 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.266542 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230e3363-1397-4a61-92ef-3a29ae22ea13-kube-api-access-98h2l" pod "230e3363-1397-4a61-92ef-3a29ae22ea13" (UID: "230e3363-1397-4a61-92ef-3a29ae22ea13"). InnerVolumeSpecName "kube-api-access-98h2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:46.266670 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.266571 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-serving-cert" pod "230e3363-1397-4a61-92ef-3a29ae22ea13" (UID: "230e3363-1397-4a61-92ef-3a29ae22ea13"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:46.266670 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.266584 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-oauth-config" pod "230e3363-1397-4a61-92ef-3a29ae22ea13" (UID: "230e3363-1397-4a61-92ef-3a29ae22ea13"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:46.365140 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.365114 4456 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:46.365140 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.365138 4456 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-serving-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:46.365284 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.365147 4456 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/230e3363-1397-4a61-92ef-3a29ae22ea13-console-oauth-config\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:46.365284 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.365159 4456 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-oauth-serving-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:46.365284 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.365169 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98h2l\" (UniqueName: \"kubernetes.io/projected/230e3363-1397-4a61-92ef-3a29ae22ea13-kube-api-access-98h2l\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:46.365284 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.365177 4456 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/230e3363-1397-4a61-92ef-3a29ae22ea13-console-config\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:46.896942 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.896901 4456 generic.go:356] "Generic (PLEG): container finished" podID="230e3363-1397-4a61-92ef-3a29ae22ea13" containerID="3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b" exitCode=0 Oct 31 02:56:46.897398 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.896967 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99cc849dd-kvzv9" Oct 31 02:56:46.897398 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.896987 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99cc849dd-kvzv9" event={"ID":"230e3363-1397-4a61-92ef-3a29ae22ea13","Type":"ContainerDied","Data":"3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b"} Oct 31 02:56:46.897398 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.897056 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99cc849dd-kvzv9" event={"ID":"230e3363-1397-4a61-92ef-3a29ae22ea13","Type":"ContainerDied","Data":"2681ce5dfd98415a7a609b37bf979235fd4bdae1d27cd5dacfc501fa392b105a"} Oct 31 02:56:46.897398 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.897077 4456 scope.go:122] "RemoveContainer" containerID="3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b" Oct 31 02:56:46.905016 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.904992 4456 scope.go:122] "RemoveContainer" containerID="3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b" Oct 31 02:56:46.905359 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:56:46.905339 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b\": container with ID starting with 3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b not found: ID does not exist" containerID="3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b" Oct 31 02:56:46.905437 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:46.905376 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b"} err="failed to get container status \"3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b\": rpc error: code = NotFound desc = could not find container \"3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b\": container with ID starting with 3810c4822d543694e72b3bbd756fe3cebffac6438c41a5e8460cb48f6886fe2b not found: ID does not exist" Oct 31 02:56:47.008187 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:47.008153 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-99cc849dd-kvzv9"] Oct 31 02:56:47.022254 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:47.022228 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-99cc849dd-kvzv9"] Oct 31 02:56:48.432861 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:48.432822 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230e3363-1397-4a61-92ef-3a29ae22ea13" path="/var/lib/kubelet/pods/230e3363-1397-4a61-92ef-3a29ae22ea13/volumes" Oct 31 02:56:55.349000 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:55.348967 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:55.349412 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:55.349028 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:55.353559 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:55.353537 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:55.937242 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:55.937213 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:56:56.073120 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:56:56.073087 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c49b66d86-zjs5z"] Oct 31 02:57:02.132423 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:57:02.132386 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 02:57:21.092197 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.092124 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c49b66d86-zjs5z" podUID="6e12e21d-4512-4211-b710-b6b6048622d2" containerName="console" containerID="cri-o://c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309" gracePeriod=15 Oct 31 02:57:21.334065 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.334020 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:57:21.370444 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.370370 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-trusted-ca-bundle\") pod \"6e12e21d-4512-4211-b710-b6b6048622d2\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " Oct 31 02:57:21.370444 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.370419 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e12e21d-4512-4211-b710-b6b6048622d2-kube-api-access-wfghh\" (UniqueName: \"kubernetes.io/projected/6e12e21d-4512-4211-b710-b6b6048622d2-kube-api-access-wfghh\") pod \"6e12e21d-4512-4211-b710-b6b6048622d2\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " Oct 31 02:57:21.370658 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.370482 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e12e21d-4512-4211-b710-b6b6048622d2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e12e21d-4512-4211-b710-b6b6048622d2-tmp\") pod \"6e12e21d-4512-4211-b710-b6b6048622d2\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " Oct 31 02:57:21.370658 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.370515 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-oauth-serving-cert\") pod \"6e12e21d-4512-4211-b710-b6b6048622d2\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " Oct 31 02:57:21.370658 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.370590 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-serving-cert\") pod \"6e12e21d-4512-4211-b710-b6b6048622d2\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " Oct 31 02:57:21.370658 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.370626 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-console-config\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-console-config\") pod \"6e12e21d-4512-4211-b710-b6b6048622d2\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " Oct 31 02:57:21.370658 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.370653 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-oauth-config\") pod \"6e12e21d-4512-4211-b710-b6b6048622d2\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " Oct 31 02:57:21.370915 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.370704 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-service-ca\") pod \"6e12e21d-4512-4211-b710-b6b6048622d2\" (UID: \"6e12e21d-4512-4211-b710-b6b6048622d2\") " Oct 31 02:57:21.370978 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.370918 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-trusted-ca-bundle" pod "6e12e21d-4512-4211-b710-b6b6048622d2" (UID: "6e12e21d-4512-4211-b710-b6b6048622d2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:57:21.371069 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.370993 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e12e21d-4512-4211-b710-b6b6048622d2-tmp" pod "6e12e21d-4512-4211-b710-b6b6048622d2" (UID: "6e12e21d-4512-4211-b710-b6b6048622d2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:57:21.371282 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.371254 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-service-ca" pod "6e12e21d-4512-4211-b710-b6b6048622d2" (UID: "6e12e21d-4512-4211-b710-b6b6048622d2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:57:21.371561 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.371541 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-oauth-serving-cert" pod "6e12e21d-4512-4211-b710-b6b6048622d2" (UID: "6e12e21d-4512-4211-b710-b6b6048622d2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:57:21.371659 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.371538 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-console-config" pod "6e12e21d-4512-4211-b710-b6b6048622d2" (UID: "6e12e21d-4512-4211-b710-b6b6048622d2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:57:21.372795 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.372773 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-serving-cert" pod "6e12e21d-4512-4211-b710-b6b6048622d2" (UID: "6e12e21d-4512-4211-b710-b6b6048622d2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:57:21.373098 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.373077 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e12e21d-4512-4211-b710-b6b6048622d2-kube-api-access-wfghh" pod "6e12e21d-4512-4211-b710-b6b6048622d2" (UID: "6e12e21d-4512-4211-b710-b6b6048622d2"). InnerVolumeSpecName "kube-api-access-wfghh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:57:21.373237 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.373207 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-oauth-config" pod "6e12e21d-4512-4211-b710-b6b6048622d2" (UID: "6e12e21d-4512-4211-b710-b6b6048622d2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:57:21.471900 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.471871 4456 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-serving-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:57:21.471900 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.471897 4456 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-console-config\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:57:21.471900 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.471907 4456 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e12e21d-4512-4211-b710-b6b6048622d2-console-oauth-config\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:57:21.472196 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.471915 4456 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:57:21.472196 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.471933 4456 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-trusted-ca-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:57:21.472196 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.471942 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfghh\" (UniqueName: \"kubernetes.io/projected/6e12e21d-4512-4211-b710-b6b6048622d2-kube-api-access-wfghh\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:57:21.472196 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.471952 4456 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e12e21d-4512-4211-b710-b6b6048622d2-tmp\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:57:21.472196 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:21.471961 4456 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e12e21d-4512-4211-b710-b6b6048622d2-oauth-serving-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:57:22.051677 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:22.051638 4456 generic.go:356] "Generic (PLEG): container finished" podID="6e12e21d-4512-4211-b710-b6b6048622d2" containerID="c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309" exitCode=0 Oct 31 02:57:22.051847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:22.051691 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c49b66d86-zjs5z" Oct 31 02:57:22.051847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:22.051709 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c49b66d86-zjs5z" event={"ID":"6e12e21d-4512-4211-b710-b6b6048622d2","Type":"ContainerDied","Data":"c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309"} Oct 31 02:57:22.051847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:22.051756 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c49b66d86-zjs5z" event={"ID":"6e12e21d-4512-4211-b710-b6b6048622d2","Type":"ContainerDied","Data":"b9cffbf23d55987d7d7ca0bfd18b3b85f7d525fd640ee6520f35b80535cdb01c"} Oct 31 02:57:22.051847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:22.051785 4456 scope.go:122] "RemoveContainer" containerID="c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309" Oct 31 02:57:22.059939 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:22.059914 4456 scope.go:122] "RemoveContainer" containerID="c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309" Oct 31 02:57:22.060206 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:57:22.060185 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309\": container with ID starting with c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309 not found: ID does not exist" containerID="c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309" Oct 31 02:57:22.060273 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:22.060213 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309"} err="failed to get container status \"c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309\": rpc error: code = NotFound desc = could not find container \"c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309\": container with ID starting with c12f6dceaf2d423c63b0fba430675e4d15f9376723b8d7ef518c9bfa06da1309 not found: ID does not exist" Oct 31 02:57:22.076958 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:22.076932 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c49b66d86-zjs5z"] Oct 31 02:57:22.084709 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:22.084687 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c49b66d86-zjs5z"] Oct 31 02:57:22.434560 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:22.434518 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e12e21d-4512-4211-b710-b6b6048622d2" path="/var/lib/kubelet/pods/6e12e21d-4512-4211-b710-b6b6048622d2/volumes" Oct 31 02:57:32.131523 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:57:32.131449 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 02:57:51.957414 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:51.957381 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6769478768-6p8zj"] Oct 31 02:57:51.957904 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:51.957704 4456 cpu_manager.go:414] "containerMap: removing container" podUID="6e12e21d-4512-4211-b710-b6b6048622d2" containerName="console" Oct 31 02:57:51.957904 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:51.957715 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e12e21d-4512-4211-b710-b6b6048622d2" containerName="console" Oct 31 02:57:51.957904 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:51.957742 4456 cpu_manager.go:414] "containerMap: removing container" podUID="230e3363-1397-4a61-92ef-3a29ae22ea13" containerName="console" Oct 31 02:57:51.957904 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:51.957747 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230e3363-1397-4a61-92ef-3a29ae22ea13" containerName="console" Oct 31 02:57:51.957904 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:51.957792 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e12e21d-4512-4211-b710-b6b6048622d2" containerName="console" Oct 31 02:57:51.957904 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:51.957804 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="230e3363-1397-4a61-92ef-3a29ae22ea13" containerName="console" Oct 31 02:57:51.960935 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:51.960918 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:51.970895 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:51.970873 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6769478768-6p8zj"] Oct 31 02:57:52.119305 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.119271 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6516fb64-f047-458c-83d8-8a3c824aed5d-console-oauth-config\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.119305 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.119310 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6516fb64-f047-458c-83d8-8a3c824aed5d-tmp\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.119538 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.119327 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-trusted-ca-bundle\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.119538 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.119389 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6516fb64-f047-458c-83d8-8a3c824aed5d-console-serving-cert\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.119538 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.119427 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-service-ca\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.119538 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.119491 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-console-config\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.119538 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.119513 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-oauth-serving-cert\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.119732 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.119612 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cpb6\" (UniqueName: \"kubernetes.io/projected/6516fb64-f047-458c-83d8-8a3c824aed5d-kube-api-access-4cpb6\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.220599 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.220512 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6516fb64-f047-458c-83d8-8a3c824aed5d-console-oauth-config\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.220599 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.220555 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6516fb64-f047-458c-83d8-8a3c824aed5d-tmp\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.220599 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.220573 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-trusted-ca-bundle\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.220818 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.220609 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6516fb64-f047-458c-83d8-8a3c824aed5d-console-serving-cert\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.220818 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.220712 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-service-ca\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.220818 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.220756 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-console-config\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.220818 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.220782 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-oauth-serving-cert\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.221012 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.220829 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cpb6\" (UniqueName: \"kubernetes.io/projected/6516fb64-f047-458c-83d8-8a3c824aed5d-kube-api-access-4cpb6\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.221104 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.221016 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6516fb64-f047-458c-83d8-8a3c824aed5d-tmp\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.221521 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.221490 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-service-ca\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.221773 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.221536 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-trusted-ca-bundle\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.221773 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.221636 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-console-config\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.221910 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.221891 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6516fb64-f047-458c-83d8-8a3c824aed5d-oauth-serving-cert\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.223101 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.223083 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6516fb64-f047-458c-83d8-8a3c824aed5d-console-serving-cert\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.223148 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.223107 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6516fb64-f047-458c-83d8-8a3c824aed5d-console-oauth-config\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.234120 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.234094 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cpb6\" (UniqueName: \"kubernetes.io/projected/6516fb64-f047-458c-83d8-8a3c824aed5d-kube-api-access-4cpb6\") pod \"console-6769478768-6p8zj\" (UID: \"6516fb64-f047-458c-83d8-8a3c824aed5d\") " pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.273100 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.273077 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:57:52.544524 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:52.544501 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6769478768-6p8zj"] Oct 31 02:57:52.546924 ip-10-0-129-107 kubenswrapper[4456]: W1031 02:57:52.546898 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6516fb64_f047_458c_83d8_8a3c824aed5d.slice/crio-57c5c11c099e57b5dce59ae0bbe94ddf7ebd00109e7c7d8cc83c61d8116870d9 WatchSource:0}: Error finding container 57c5c11c099e57b5dce59ae0bbe94ddf7ebd00109e7c7d8cc83c61d8116870d9: Status 404 returned error can't find the container with id 57c5c11c099e57b5dce59ae0bbe94ddf7ebd00109e7c7d8cc83c61d8116870d9 Oct 31 02:57:53.168573 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:53.168532 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6769478768-6p8zj" event={"ID":"6516fb64-f047-458c-83d8-8a3c824aed5d","Type":"ContainerStarted","Data":"411d607e675989b6a5b03360cb80345533c94b0cc617efbf9789b2f8225fc42f"} Oct 31 02:57:53.168915 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:53.168594 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6769478768-6p8zj" event={"ID":"6516fb64-f047-458c-83d8-8a3c824aed5d","Type":"ContainerStarted","Data":"57c5c11c099e57b5dce59ae0bbe94ddf7ebd00109e7c7d8cc83c61d8116870d9"} Oct 31 02:57:53.195206 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:57:53.195164 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6769478768-6p8zj" podStartSLOduration=2.195151465 podStartE2EDuration="2.195151465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:57:53.193781562 +0000 UTC m=+154.874307688" watchObservedRunningTime="2026-10-31 02:57:53.195151465 +0000 UTC m=+154.875677592" Oct 31 02:58:02.132986 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:58:02.132947 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 02:58:02.274215 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:02.274178 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:58:02.274365 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:02.274270 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:58:02.278847 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:02.278820 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:58:03.210541 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:03.210511 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6769478768-6p8zj" Oct 31 02:58:03.264938 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:03.264903 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d474bb7b-vpljv"] Oct 31 02:58:28.288691 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.288621 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-84d474bb7b-vpljv" podUID="57c6b015-ac0d-4729-af0b-258c96e6630c" containerName="console" containerID="cri-o://e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812" gracePeriod=15 Oct 31 02:58:28.528890 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.528863 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:58:28.617305 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617221 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-oauth-serving-cert\") pod \"57c6b015-ac0d-4729-af0b-258c96e6630c\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " Oct 31 02:58:28.617305 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617274 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-trusted-ca-bundle\") pod \"57c6b015-ac0d-4729-af0b-258c96e6630c\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " Oct 31 02:58:28.617527 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617311 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c6b015-ac0d-4729-af0b-258c96e6630c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c6b015-ac0d-4729-af0b-258c96e6630c-tmp\") pod \"57c6b015-ac0d-4729-af0b-258c96e6630c\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " Oct 31 02:58:28.617527 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617346 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c6b015-ac0d-4729-af0b-258c96e6630c-kube-api-access-p5949\" (UniqueName: \"kubernetes.io/projected/57c6b015-ac0d-4729-af0b-258c96e6630c-kube-api-access-p5949\") pod \"57c6b015-ac0d-4729-af0b-258c96e6630c\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " Oct 31 02:58:28.617527 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617377 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-serving-cert\") pod \"57c6b015-ac0d-4729-af0b-258c96e6630c\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " Oct 31 02:58:28.617527 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617400 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-oauth-config\") pod \"57c6b015-ac0d-4729-af0b-258c96e6630c\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " Oct 31 02:58:28.617527 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617417 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-console-config\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-console-config\") pod \"57c6b015-ac0d-4729-af0b-258c96e6630c\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " Oct 31 02:58:28.617527 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617456 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-service-ca\") pod \"57c6b015-ac0d-4729-af0b-258c96e6630c\" (UID: \"57c6b015-ac0d-4729-af0b-258c96e6630c\") " Oct 31 02:58:28.617864 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617576 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c6b015-ac0d-4729-af0b-258c96e6630c-tmp" pod "57c6b015-ac0d-4729-af0b-258c96e6630c" (UID: "57c6b015-ac0d-4729-af0b-258c96e6630c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:58:28.617864 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617738 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-oauth-serving-cert" pod "57c6b015-ac0d-4729-af0b-258c96e6630c" (UID: "57c6b015-ac0d-4729-af0b-258c96e6630c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:58:28.617864 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617786 4456 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c6b015-ac0d-4729-af0b-258c96e6630c-tmp\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:28.618009 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617952 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-trusted-ca-bundle" pod "57c6b015-ac0d-4729-af0b-258c96e6630c" (UID: "57c6b015-ac0d-4729-af0b-258c96e6630c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:58:28.618009 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617984 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-service-ca" pod "57c6b015-ac0d-4729-af0b-258c96e6630c" (UID: "57c6b015-ac0d-4729-af0b-258c96e6630c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:58:28.618009 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.617987 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-console-config" pod "57c6b015-ac0d-4729-af0b-258c96e6630c" (UID: "57c6b015-ac0d-4729-af0b-258c96e6630c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:58:28.619594 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.619562 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c6b015-ac0d-4729-af0b-258c96e6630c-kube-api-access-p5949" pod "57c6b015-ac0d-4729-af0b-258c96e6630c" (UID: "57c6b015-ac0d-4729-af0b-258c96e6630c"). InnerVolumeSpecName "kube-api-access-p5949". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:58:28.619704 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.619622 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-oauth-config" pod "57c6b015-ac0d-4729-af0b-258c96e6630c" (UID: "57c6b015-ac0d-4729-af0b-258c96e6630c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:58:28.619704 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.619644 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-serving-cert" pod "57c6b015-ac0d-4729-af0b-258c96e6630c" (UID: "57c6b015-ac0d-4729-af0b-258c96e6630c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:58:28.718995 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.718947 4456 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-oauth-serving-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:28.718995 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.718985 4456 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-trusted-ca-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:28.718995 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.718998 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5949\" (UniqueName: \"kubernetes.io/projected/57c6b015-ac0d-4729-af0b-258c96e6630c-kube-api-access-p5949\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:28.719269 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.719012 4456 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-serving-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:28.719269 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.719024 4456 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57c6b015-ac0d-4729-af0b-258c96e6630c-console-oauth-config\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:28.719269 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.719064 4456 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-console-config\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:28.719269 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:28.719077 4456 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57c6b015-ac0d-4729-af0b-258c96e6630c-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:29.306357 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:29.306325 4456 generic.go:356] "Generic (PLEG): container finished" podID="57c6b015-ac0d-4729-af0b-258c96e6630c" containerID="e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812" exitCode=0 Oct 31 02:58:29.306783 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:29.306372 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d474bb7b-vpljv" Oct 31 02:58:29.306783 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:29.306385 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d474bb7b-vpljv" event={"ID":"57c6b015-ac0d-4729-af0b-258c96e6630c","Type":"ContainerDied","Data":"e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812"} Oct 31 02:58:29.306783 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:29.306433 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d474bb7b-vpljv" event={"ID":"57c6b015-ac0d-4729-af0b-258c96e6630c","Type":"ContainerDied","Data":"25827de4f8d53392c001c03567256266f3256d41f125c8af865a56e67b1e2a02"} Oct 31 02:58:29.306783 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:29.306455 4456 scope.go:122] "RemoveContainer" containerID="e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812" Oct 31 02:58:29.319591 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:29.319570 4456 scope.go:122] "RemoveContainer" containerID="e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812" Oct 31 02:58:29.319850 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:58:29.319830 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812\": container with ID starting with e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812 not found: ID does not exist" containerID="e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812" Oct 31 02:58:29.319900 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:29.319858 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812"} err="failed to get container status \"e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812\": rpc error: code = NotFound desc = could not find container \"e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812\": container with ID starting with e022540bc2fd3600c04aa12447600c5b9ec9736baeb32d3c8d37d2457cd66812 not found: ID does not exist" Oct 31 02:58:29.333028 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:29.333003 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d474bb7b-vpljv"] Oct 31 02:58:29.338667 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:29.338645 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84d474bb7b-vpljv"] Oct 31 02:58:30.432466 ip-10-0-129-107 kubenswrapper[4456]: I1031 02:58:30.432420 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c6b015-ac0d-4729-af0b-258c96e6630c" path="/var/lib/kubelet/pods/57c6b015-ac0d-4729-af0b-258c96e6630c/volumes" Oct 31 02:58:32.131825 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:58:32.131792 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 02:59:02.132526 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:59:02.132443 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 02:59:32.132892 ip-10-0-129-107 kubenswrapper[4456]: E1031 02:59:32.132854 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:00:02.133184 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:00:02.133147 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:00:10.758488 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.758454 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt"] Oct 31 03:00:10.758934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.758781 4456 cpu_manager.go:414] "containerMap: removing container" podUID="57c6b015-ac0d-4729-af0b-258c96e6630c" containerName="console" Oct 31 03:00:10.758934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.758793 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c6b015-ac0d-4729-af0b-258c96e6630c" containerName="console" Oct 31 03:00:10.758934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.758850 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c6b015-ac0d-4729-af0b-258c96e6630c" containerName="console" Oct 31 03:00:10.761822 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.761804 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:10.767560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.767535 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 03:00:10.769484 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.769465 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-qwqdn\"" Oct 31 03:00:10.770182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.770161 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 03:00:10.771693 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.771676 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt"] Oct 31 03:00:10.838406 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.838371 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:10.838595 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.838415 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47l8\" (UniqueName: \"kubernetes.io/projected/13ae11bf-a60a-4f89-b078-ede5f7a80788-kube-api-access-x47l8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:10.838595 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.838542 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:10.939397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.939361 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47l8\" (UniqueName: \"kubernetes.io/projected/13ae11bf-a60a-4f89-b078-ede5f7a80788-kube-api-access-x47l8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:10.939525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.939422 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:10.939525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.939470 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:10.939732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.939715 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:10.939769 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.939746 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:10.953145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:10.953115 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47l8\" (UniqueName: \"kubernetes.io/projected/13ae11bf-a60a-4f89-b078-ede5f7a80788-kube-api-access-x47l8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:11.083813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:11.083739 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:00:11.355576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:11.355552 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt"] Oct 31 03:00:11.357403 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:00:11.357374 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ae11bf_a60a_4f89_b078_ede5f7a80788.slice/crio-ea0c77743943e4a8023ec51d8dc097d0afa63d1f8545f15f561fdd87aaabc7a7 WatchSource:0}: Error finding container ea0c77743943e4a8023ec51d8dc097d0afa63d1f8545f15f561fdd87aaabc7a7: Status 404 returned error can't find the container with id ea0c77743943e4a8023ec51d8dc097d0afa63d1f8545f15f561fdd87aaabc7a7 Oct 31 03:00:11.665352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:11.665312 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" event={"ID":"13ae11bf-a60a-4f89-b078-ede5f7a80788","Type":"ContainerStarted","Data":"ea0c77743943e4a8023ec51d8dc097d0afa63d1f8545f15f561fdd87aaabc7a7"} Oct 31 03:00:12.409582 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.409547 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9"] Oct 31 03:00:12.413300 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.413274 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.424764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.424735 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9"] Oct 31 03:00:12.424894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.424853 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-4j8rv\"" Oct 31 03:00:12.453737 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.453693 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86fz\" (UniqueName: \"kubernetes.io/projected/96c66979-ebac-48ab-bc68-431eb8d495b1-kube-api-access-d86fz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.453892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.453762 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.453892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.453815 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.506404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.506371 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb"] Oct 31 03:00:12.509859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.509838 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.519245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.519226 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb"] Oct 31 03:00:12.521255 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.521235 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-s7fvx\"" Oct 31 03:00:12.554660 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.554628 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4ld\" (UniqueName: \"kubernetes.io/projected/3657424b-e958-4a2a-aed9-90c659954a62-kube-api-access-gz4ld\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.554851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.554683 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86fz\" (UniqueName: \"kubernetes.io/projected/96c66979-ebac-48ab-bc68-431eb8d495b1-kube-api-access-d86fz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.554851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.554760 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.554851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.554803 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.554989 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.554951 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.555127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.555001 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.555372 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.555349 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.555459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.555373 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.568869 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.568837 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86fz\" (UniqueName: \"kubernetes.io/projected/96c66979-ebac-48ab-bc68-431eb8d495b1-kube-api-access-d86fz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.605554 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.605520 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw"] Oct 31 03:00:12.609375 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.609348 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:12.616897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.616871 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-29hnj\"" Oct 31 03:00:12.618739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.618713 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw"] Oct 31 03:00:12.655645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.655609 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.655785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.655653 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.655785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.655696 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:12.655785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.655739 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzgg\" (UniqueName: \"kubernetes.io/projected/a7857fff-dff2-4b03-b39d-aec834cff058-kube-api-access-xpzgg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:12.655785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.655771 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:12.656016 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.655848 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4ld\" (UniqueName: \"kubernetes.io/projected/3657424b-e958-4a2a-aed9-90c659954a62-kube-api-access-gz4ld\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.656016 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.655958 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.656158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.656077 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.669697 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.669647 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4ld\" (UniqueName: \"kubernetes.io/projected/3657424b-e958-4a2a-aed9-90c659954a62-kube-api-access-gz4ld\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.713621 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.713586 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk"] Oct 31 03:00:12.717380 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.717361 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:12.724808 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.724785 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-48c4r\"" Oct 31 03:00:12.726390 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.726354 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk"] Oct 31 03:00:12.733561 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.733537 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:00:12.757931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.756745 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:12.757931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.756798 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:12.757931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.756836 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:12.757931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.756862 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4pzq\" (UniqueName: \"kubernetes.io/projected/b8039e8a-b518-4a48-94ce-a94333112d30-kube-api-access-t4pzq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:12.757931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.756884 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzgg\" (UniqueName: \"kubernetes.io/projected/a7857fff-dff2-4b03-b39d-aec834cff058-kube-api-access-xpzgg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:12.757931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.756910 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:12.757931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.757352 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:12.757931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.757627 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:12.771707 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.771680 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzgg\" (UniqueName: \"kubernetes.io/projected/a7857fff-dff2-4b03-b39d-aec834cff058-kube-api-access-xpzgg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:12.830059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.830004 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:00:12.857859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.857419 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4pzq\" (UniqueName: \"kubernetes.io/projected/b8039e8a-b518-4a48-94ce-a94333112d30-kube-api-access-t4pzq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:12.857859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.857546 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:12.857859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.857602 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:12.858022 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.857924 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:12.858022 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.858008 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:12.871410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.871384 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4pzq\" (UniqueName: \"kubernetes.io/projected/b8039e8a-b518-4a48-94ce-a94333112d30-kube-api-access-t4pzq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:12.931267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:12.931203 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:00:13.024368 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.024221 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9"] Oct 31 03:00:13.026693 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:00:13.026662 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c66979_ebac_48ab_bc68_431eb8d495b1.slice/crio-c307da743cdc7a92b194ae45b24f76e41ff9773bef7993a22362ce5ac591ea62 WatchSource:0}: Error finding container c307da743cdc7a92b194ae45b24f76e41ff9773bef7993a22362ce5ac591ea62: Status 404 returned error can't find the container with id c307da743cdc7a92b194ae45b24f76e41ff9773bef7993a22362ce5ac591ea62 Oct 31 03:00:13.036762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.036734 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:00:13.120932 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.120850 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb"] Oct 31 03:00:13.213114 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.213081 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw"] Oct 31 03:00:13.215502 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:00:13.215469 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7857fff_dff2_4b03_b39d_aec834cff058.slice/crio-fd4eaec01de32e93a8a2b0735719f9abd1cafc3d1a51922372ef656b1c289764 WatchSource:0}: Error finding container fd4eaec01de32e93a8a2b0735719f9abd1cafc3d1a51922372ef656b1c289764: Status 404 returned error can't find the container with id fd4eaec01de32e93a8a2b0735719f9abd1cafc3d1a51922372ef656b1c289764 Oct 31 03:00:13.331520 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.331479 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk"] Oct 31 03:00:13.595710 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.595635 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2"] Oct 31 03:00:13.599443 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.599419 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:13.611676 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.611651 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2"] Oct 31 03:00:13.611846 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.611824 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-bfgxh\"" Oct 31 03:00:13.664453 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.664416 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:13.664630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.664553 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:13.664630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.664583 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxdkk\" (UniqueName: \"kubernetes.io/projected/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-kube-api-access-qxdkk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:13.676048 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.675998 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" event={"ID":"a7857fff-dff2-4b03-b39d-aec834cff058","Type":"ContainerStarted","Data":"fd4eaec01de32e93a8a2b0735719f9abd1cafc3d1a51922372ef656b1c289764"} Oct 31 03:00:13.677457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.677419 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" event={"ID":"96c66979-ebac-48ab-bc68-431eb8d495b1","Type":"ContainerStarted","Data":"c307da743cdc7a92b194ae45b24f76e41ff9773bef7993a22362ce5ac591ea62"} Oct 31 03:00:13.678733 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.678700 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" event={"ID":"b8039e8a-b518-4a48-94ce-a94333112d30","Type":"ContainerStarted","Data":"c26101f80fa3006e9a1800a4c607dc21e28e81f2bcca343878dad18f73a11d3e"} Oct 31 03:00:13.680008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.679989 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" event={"ID":"3657424b-e958-4a2a-aed9-90c659954a62","Type":"ContainerStarted","Data":"dcb5a24a164f88e5b28b4878f95f85dafafa3a80396dd40e872f6666b48e36ac"} Oct 31 03:00:13.765723 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.765685 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:13.765886 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.765794 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:13.765886 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.765828 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxdkk\" (UniqueName: \"kubernetes.io/projected/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-kube-api-access-qxdkk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:13.766181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.766154 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:13.766530 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.766480 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:13.780956 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.780926 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxdkk\" (UniqueName: \"kubernetes.io/projected/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-kube-api-access-qxdkk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:13.920207 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:13.920172 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:14.222441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:14.222404 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2"] Oct 31 03:00:14.225652 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:00:14.225623 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5143b6_190a_4cb9_83c8_41fdb4d1619a.slice/crio-996eb78e77b2b076c28a8ea844c457efb7ba57a08c0c10038069fa3a4df9eff1 WatchSource:0}: Error finding container 996eb78e77b2b076c28a8ea844c457efb7ba57a08c0c10038069fa3a4df9eff1: Status 404 returned error can't find the container with id 996eb78e77b2b076c28a8ea844c457efb7ba57a08c0c10038069fa3a4df9eff1 Oct 31 03:00:14.687658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:14.687624 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" event={"ID":"4a5143b6-190a-4cb9-83c8-41fdb4d1619a","Type":"ContainerStarted","Data":"996eb78e77b2b076c28a8ea844c457efb7ba57a08c0c10038069fa3a4df9eff1"} Oct 31 03:00:15.144677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.144594 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8"] Oct 31 03:00:15.149098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.149068 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.157953 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.157916 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-b6h95\"" Oct 31 03:00:15.159791 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.159769 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8"] Oct 31 03:00:15.183416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.183066 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.183416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.183126 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.183416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.183213 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6c44\" (UniqueName: \"kubernetes.io/projected/b21ae90f-021c-45a3-ab03-7ec953df43ab-kube-api-access-q6c44\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.284780 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.284738 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6c44\" (UniqueName: \"kubernetes.io/projected/b21ae90f-021c-45a3-ab03-7ec953df43ab-kube-api-access-q6c44\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.284963 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.284863 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.284963 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.284909 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.285699 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.285376 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.285972 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.285905 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.305508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.305463 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6c44\" (UniqueName: \"kubernetes.io/projected/b21ae90f-021c-45a3-ab03-7ec953df43ab-kube-api-access-q6c44\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.473308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.473084 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:15.816058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:15.815959 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8"] Oct 31 03:00:15.824748 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:00:15.824715 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21ae90f_021c_45a3_ab03_7ec953df43ab.slice/crio-aa8ff5b94782542ad6f3e8fd7fee22946bc56bb7a02681af9f9ac4462d1a71f6 WatchSource:0}: Error finding container aa8ff5b94782542ad6f3e8fd7fee22946bc56bb7a02681af9f9ac4462d1a71f6: Status 404 returned error can't find the container with id aa8ff5b94782542ad6f3e8fd7fee22946bc56bb7a02681af9f9ac4462d1a71f6 Oct 31 03:00:16.185072 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.184776 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z"] Oct 31 03:00:16.191061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.190396 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.202969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.202481 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 03:00:16.202969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.202514 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zfwtw\"" Oct 31 03:00:16.202969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.202591 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 03:00:16.202969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.202602 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 03:00:16.208929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.208902 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z"] Oct 31 03:00:16.300217 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.299703 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.300217 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.299751 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.300217 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.299780 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.300217 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.299820 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.300217 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.299851 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5wd8\" (UniqueName: \"kubernetes.io/projected/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-kube-api-access-t5wd8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.300217 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.299906 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.300217 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.299957 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.300217 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.299990 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.300217 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.300016 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.401092 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.401145 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.401174 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.401252 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.401282 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.401307 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.401344 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.401433 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5wd8\" (UniqueName: \"kubernetes.io/projected/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-kube-api-access-t5wd8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.401486 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.401975 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.402630 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.402899 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.403217 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.403677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.403615 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.414403 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.414339 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.420071 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.420008 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.430409 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.430362 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5wd8\" (UniqueName: \"kubernetes.io/projected/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-kube-api-access-t5wd8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.438074 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.437914 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef33ec8e-50ff-4f0b-a84c-25a9b07c2221-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7r8z\" (UID: \"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.509862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.509832 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:16.722495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.722312 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" event={"ID":"b21ae90f-021c-45a3-ab03-7ec953df43ab","Type":"ContainerStarted","Data":"aa8ff5b94782542ad6f3e8fd7fee22946bc56bb7a02681af9f9ac4462d1a71f6"} Oct 31 03:00:16.844852 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:16.844815 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z"] Oct 31 03:00:16.854745 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:00:16.854704 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef33ec8e_50ff_4f0b_a84c_25a9b07c2221.slice/crio-347b48a34ac05138fb94468e2daafbfe00912221eaabb01a25819376af9de322 WatchSource:0}: Error finding container 347b48a34ac05138fb94468e2daafbfe00912221eaabb01a25819376af9de322: Status 404 returned error can't find the container with id 347b48a34ac05138fb94468e2daafbfe00912221eaabb01a25819376af9de322 Oct 31 03:00:17.732975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:17.732875 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" event={"ID":"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221","Type":"ContainerStarted","Data":"347b48a34ac05138fb94468e2daafbfe00912221eaabb01a25819376af9de322"} Oct 31 03:00:17.789881 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:17.789362 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc"] Oct 31 03:00:17.796218 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:17.795513 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:17.806927 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:17.806893 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc"] Oct 31 03:00:17.915609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:17.915569 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:17.916121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:17.915681 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mwch\" (UniqueName: \"kubernetes.io/projected/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-kube-api-access-4mwch\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:17.916121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:17.915738 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:18.016864 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:18.016774 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:18.016864 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:18.016847 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:18.017118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:18.016936 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mwch\" (UniqueName: \"kubernetes.io/projected/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-kube-api-access-4mwch\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:18.017718 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:18.017685 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:18.018027 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:18.018007 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:18.052408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:18.052377 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mwch\" (UniqueName: \"kubernetes.io/projected/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-kube-api-access-4mwch\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:18.123983 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:18.123713 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:00:18.471085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:18.471058 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc"] Oct 31 03:00:18.474397 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:00:18.474308 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd99703_7989_4cd3_a014_d6c8f4ebc4e2.slice/crio-fd9b1e87bf84261e12ed464142c8b10ed475a3ad724621815def0b3d9f98c6c6 WatchSource:0}: Error finding container fd9b1e87bf84261e12ed464142c8b10ed475a3ad724621815def0b3d9f98c6c6: Status 404 returned error can't find the container with id fd9b1e87bf84261e12ed464142c8b10ed475a3ad724621815def0b3d9f98c6c6 Oct 31 03:00:18.749064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:18.748918 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" event={"ID":"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2","Type":"ContainerStarted","Data":"fd9b1e87bf84261e12ed464142c8b10ed475a3ad724621815def0b3d9f98c6c6"} Oct 31 03:00:20.587492 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.587451 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l"] Oct 31 03:00:20.592267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.592244 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.603005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.602527 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l"] Oct 31 03:00:20.630098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.628086 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww"] Oct 31 03:00:20.632822 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.632798 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.643295 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.643273 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww"] Oct 31 03:00:20.649562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.649521 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjbm2\" (UniqueName: \"kubernetes.io/projected/aa862741-8e25-486a-a803-e5c617f6dd0d-kube-api-access-mjbm2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.649679 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.649648 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.649750 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.649678 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.663055 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.662995 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6"] Oct 31 03:00:20.667246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.667227 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:20.679124 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.679100 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6"] Oct 31 03:00:20.704105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.703706 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d"] Oct 31 03:00:20.708155 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.708131 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:20.718799 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.718773 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d"] Oct 31 03:00:20.752079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.750720 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.752079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.750765 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:20.752079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.750840 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:20.752079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.750876 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pcxn\" (UniqueName: \"kubernetes.io/projected/b3e014e7-5988-443b-9543-45e65feef843-kube-api-access-9pcxn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.752079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.750939 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.752079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.750971 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.752079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.751058 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.752079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.751090 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmdpj\" (UniqueName: \"kubernetes.io/projected/6f7a4303-69ba-4f48-916c-1b9b99f477e6-kube-api-access-rmdpj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:20.752079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.751133 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjbm2\" (UniqueName: \"kubernetes.io/projected/aa862741-8e25-486a-a803-e5c617f6dd0d-kube-api-access-mjbm2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.753604 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.753491 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.753820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.753779 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.769106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.769057 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjbm2\" (UniqueName: \"kubernetes.io/projected/aa862741-8e25-486a-a803-e5c617f6dd0d-kube-api-access-mjbm2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.852330 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.852240 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:20.852330 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.852298 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.852330 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.852330 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:20.852684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.852371 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfff9\" (UniqueName: \"kubernetes.io/projected/ff332518-f459-4504-813f-b64fabf4001a-kube-api-access-tfff9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:20.852684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.852417 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:20.852684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.852447 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:20.852684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.852479 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pcxn\" (UniqueName: \"kubernetes.io/projected/b3e014e7-5988-443b-9543-45e65feef843-kube-api-access-9pcxn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.852684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.852555 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.852684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.852585 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmdpj\" (UniqueName: \"kubernetes.io/projected/6f7a4303-69ba-4f48-916c-1b9b99f477e6-kube-api-access-rmdpj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:20.853496 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.853468 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:20.853662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.853639 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:20.853873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.853804 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.854223 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.854200 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.870592 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.870534 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmdpj\" (UniqueName: \"kubernetes.io/projected/6f7a4303-69ba-4f48-916c-1b9b99f477e6-kube-api-access-rmdpj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:20.876939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.876878 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pcxn\" (UniqueName: \"kubernetes.io/projected/b3e014e7-5988-443b-9543-45e65feef843-kube-api-access-9pcxn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.922438 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.922178 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:00:20.953830 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.953797 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:20.953965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.953870 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfff9\" (UniqueName: \"kubernetes.io/projected/ff332518-f459-4504-813f-b64fabf4001a-kube-api-access-tfff9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:20.953965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.953919 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:20.954431 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.954402 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:20.954532 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.954412 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:20.965404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.965182 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:00:20.980399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.980363 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfff9\" (UniqueName: \"kubernetes.io/projected/ff332518-f459-4504-813f-b64fabf4001a-kube-api-access-tfff9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:20.990387 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:20.990165 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:00:21.039664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:21.039383 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:00:21.271640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:21.271609 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww"] Oct 31 03:00:21.274413 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:00:21.274372 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e014e7_5988_443b_9543_45e65feef843.slice/crio-853d5c91a54474ade7316cead8b9cde6c002803a224fd093bc83ca4b28ccf470 WatchSource:0}: Error finding container 853d5c91a54474ade7316cead8b9cde6c002803a224fd093bc83ca4b28ccf470: Status 404 returned error can't find the container with id 853d5c91a54474ade7316cead8b9cde6c002803a224fd093bc83ca4b28ccf470 Oct 31 03:00:21.288537 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:21.287460 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l"] Oct 31 03:00:21.356361 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:21.355379 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6"] Oct 31 03:00:21.395314 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:21.395286 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d"] Oct 31 03:00:21.398985 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:00:21.398957 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff332518_f459_4504_813f_b64fabf4001a.slice/crio-d13c0dfb61ae8198d535f5d4d5cb015975a4a1c9741bd44b97ecc9f3a19046a8 WatchSource:0}: Error finding container d13c0dfb61ae8198d535f5d4d5cb015975a4a1c9741bd44b97ecc9f3a19046a8: Status 404 returned error can't find the container with id d13c0dfb61ae8198d535f5d4d5cb015975a4a1c9741bd44b97ecc9f3a19046a8 Oct 31 03:00:21.807832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:21.807790 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" event={"ID":"b3e014e7-5988-443b-9543-45e65feef843","Type":"ContainerStarted","Data":"853d5c91a54474ade7316cead8b9cde6c002803a224fd093bc83ca4b28ccf470"} Oct 31 03:00:21.813823 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:21.813791 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" event={"ID":"6f7a4303-69ba-4f48-916c-1b9b99f477e6","Type":"ContainerStarted","Data":"bcc7cf49a2e2b429ec5e3f4ff2c63ce98f61732d657e897e9dac48670835a3df"} Oct 31 03:00:21.829605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:21.829572 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" event={"ID":"aa862741-8e25-486a-a803-e5c617f6dd0d","Type":"ContainerStarted","Data":"d04bcbc002b545e5a70d00fb57e1e3a701b3c9e4f6e4e0eef53f6251b2a582e0"} Oct 31 03:00:21.832246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:21.832219 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" event={"ID":"ff332518-f459-4504-813f-b64fabf4001a","Type":"ContainerStarted","Data":"d13c0dfb61ae8198d535f5d4d5cb015975a4a1c9741bd44b97ecc9f3a19046a8"} Oct 31 03:00:27.902324 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:27.901223 4456 generic.go:356] "Generic (PLEG): container finished" podID="13ae11bf-a60a-4f89-b078-ede5f7a80788" containerID="fbdebd88052dfe1893abf435b5bdf1ec6b5711222bb0313de98598309c9bb99f" exitCode=0 Oct 31 03:00:27.902324 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:27.901372 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" event={"ID":"13ae11bf-a60a-4f89-b078-ede5f7a80788","Type":"ContainerDied","Data":"fbdebd88052dfe1893abf435b5bdf1ec6b5711222bb0313de98598309c9bb99f"} Oct 31 03:00:27.911699 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:27.911671 4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:00:28.472518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:28.472438 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:00:28.472664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:28.472540 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:00:28.472664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:28.472593 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:00:28.933606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:28.933562 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" event={"ID":"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221","Type":"ContainerStarted","Data":"bf7c53e1e5a5fd82a6c3f25fbe9d5056e5c7a4eb841ec7592388edfe57ecdd5f"} Oct 31 03:00:28.944083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:28.941223 4456 generic.go:356] "Generic (PLEG): container finished" podID="b21ae90f-021c-45a3-ab03-7ec953df43ab" containerID="2f50e70782e63ecffe3e8f19b3dc4e50cad2d9cc6b0e7d3f63f8df532232c30d" exitCode=0 Oct 31 03:00:28.944083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:28.941327 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" event={"ID":"b21ae90f-021c-45a3-ab03-7ec953df43ab","Type":"ContainerDied","Data":"2f50e70782e63ecffe3e8f19b3dc4e50cad2d9cc6b0e7d3f63f8df532232c30d"} Oct 31 03:00:28.969192 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:28.968607 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" podStartSLOduration=1.356424574 podStartE2EDuration="12.968589206s" totalImagesPullingTime="11.612164632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:00:28.964669065 +0000 UTC m=+310.645195193" watchObservedRunningTime="2026-10-31 03:00:28.968589206 +0000 UTC m=+310.649115335" Oct 31 03:00:29.511448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:29.511415 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:29.518500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:29.518202 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:29.949251 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:29.949209 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:29.950258 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:29.950232 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:00:30.967168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:30.965933 4456 generic.go:356] "Generic (PLEG): container finished" podID="b21ae90f-021c-45a3-ab03-7ec953df43ab" containerID="09b435c1afacc4537211e68c466a75ff94ba18df62ee9801cfeb55d5e5195001" exitCode=0 Oct 31 03:00:30.967168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:30.967100 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" event={"ID":"b21ae90f-021c-45a3-ab03-7ec953df43ab","Type":"ContainerDied","Data":"09b435c1afacc4537211e68c466a75ff94ba18df62ee9801cfeb55d5e5195001"} Oct 31 03:00:32.241660 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:00:32.241577 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:00:33.013338 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:33.012627 4456 generic.go:356] "Generic (PLEG): container finished" podID="0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" containerID="be081d6375fcab1d9d95f121d12f9a672ca480304fed677e4d2fe0a7edcae37d" exitCode=0 Oct 31 03:00:33.013338 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:33.012685 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" event={"ID":"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2","Type":"ContainerDied","Data":"be081d6375fcab1d9d95f121d12f9a672ca480304fed677e4d2fe0a7edcae37d"} Oct 31 03:00:34.028991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:34.028952 4456 generic.go:356] "Generic (PLEG): container finished" podID="96c66979-ebac-48ab-bc68-431eb8d495b1" containerID="891dd888a7a6193440f5ee137b50c391fa6c16bfb95d649aada8adc307f78b07" exitCode=0 Oct 31 03:00:34.028991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:34.029121 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" event={"ID":"96c66979-ebac-48ab-bc68-431eb8d495b1","Type":"ContainerDied","Data":"891dd888a7a6193440f5ee137b50c391fa6c16bfb95d649aada8adc307f78b07"} Oct 31 03:00:35.041913 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.041882 4456 generic.go:356] "Generic (PLEG): container finished" podID="4a5143b6-190a-4cb9-83c8-41fdb4d1619a" containerID="717b81e8b081f90e7b0a6e2186c04286d0d365d6a5ec9c16b11b3a8316b343ae" exitCode=0 Oct 31 03:00:35.042403 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.041967 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" event={"ID":"4a5143b6-190a-4cb9-83c8-41fdb4d1619a","Type":"ContainerDied","Data":"717b81e8b081f90e7b0a6e2186c04286d0d365d6a5ec9c16b11b3a8316b343ae"} Oct 31 03:00:35.043959 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.043936 4456 generic.go:356] "Generic (PLEG): container finished" podID="b3e014e7-5988-443b-9543-45e65feef843" containerID="1edaf7d9ddd2e6e2b46c9bf5413ae480651dcbc7afac2be8986819a01f16683a" exitCode=0 Oct 31 03:00:35.044089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.044055 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" event={"ID":"b3e014e7-5988-443b-9543-45e65feef843","Type":"ContainerDied","Data":"1edaf7d9ddd2e6e2b46c9bf5413ae480651dcbc7afac2be8986819a01f16683a"} Oct 31 03:00:35.046159 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.046133 4456 generic.go:356] "Generic (PLEG): container finished" podID="a7857fff-dff2-4b03-b39d-aec834cff058" containerID="821726783dff2481285bb8ebb1ebea667071d13d30a42ae0c5f0ba5f202b08f9" exitCode=0 Oct 31 03:00:35.046281 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.046257 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" event={"ID":"a7857fff-dff2-4b03-b39d-aec834cff058","Type":"ContainerDied","Data":"821726783dff2481285bb8ebb1ebea667071d13d30a42ae0c5f0ba5f202b08f9"} Oct 31 03:00:35.048864 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.048837 4456 generic.go:356] "Generic (PLEG): container finished" podID="6f7a4303-69ba-4f48-916c-1b9b99f477e6" containerID="fd38ccb408d9c46983d60ae5ac2f48b24f2c0bac9046e53a8af1a1a2567014ac" exitCode=0 Oct 31 03:00:35.048951 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.048907 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" event={"ID":"6f7a4303-69ba-4f48-916c-1b9b99f477e6","Type":"ContainerDied","Data":"fd38ccb408d9c46983d60ae5ac2f48b24f2c0bac9046e53a8af1a1a2567014ac"} Oct 31 03:00:35.051751 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.051725 4456 generic.go:356] "Generic (PLEG): container finished" podID="b8039e8a-b518-4a48-94ce-a94333112d30" containerID="8a970c01d1a17a44eadc2b689f902303709fb044128b725a91144fba903992db" exitCode=0 Oct 31 03:00:35.051863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.051806 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" event={"ID":"b8039e8a-b518-4a48-94ce-a94333112d30","Type":"ContainerDied","Data":"8a970c01d1a17a44eadc2b689f902303709fb044128b725a91144fba903992db"} Oct 31 03:00:35.054631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.054608 4456 generic.go:356] "Generic (PLEG): container finished" podID="3657424b-e958-4a2a-aed9-90c659954a62" containerID="f3a05a2cf955f3c18e67befa037aa2c4f6ad6e518c53d1739125a42bfda281be" exitCode=0 Oct 31 03:00:35.054703 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.054693 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" event={"ID":"3657424b-e958-4a2a-aed9-90c659954a62","Type":"ContainerDied","Data":"f3a05a2cf955f3c18e67befa037aa2c4f6ad6e518c53d1739125a42bfda281be"} Oct 31 03:00:35.057314 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.056924 4456 generic.go:356] "Generic (PLEG): container finished" podID="aa862741-8e25-486a-a803-e5c617f6dd0d" containerID="8104776d04216d8480e70fc50d664adb0f5fabc9e236d2c75f5ff827c69450d0" exitCode=0 Oct 31 03:00:35.057314 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.056997 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" event={"ID":"aa862741-8e25-486a-a803-e5c617f6dd0d","Type":"ContainerDied","Data":"8104776d04216d8480e70fc50d664adb0f5fabc9e236d2c75f5ff827c69450d0"} Oct 31 03:00:35.061197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.060364 4456 generic.go:356] "Generic (PLEG): container finished" podID="ff332518-f459-4504-813f-b64fabf4001a" containerID="8d47c11a7fdbba188e023e432839b3b4ffb03c4a627f558fbc88e5930b007856" exitCode=0 Oct 31 03:00:35.061197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:35.060400 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" event={"ID":"ff332518-f459-4504-813f-b64fabf4001a","Type":"ContainerDied","Data":"8d47c11a7fdbba188e023e432839b3b4ffb03c4a627f558fbc88e5930b007856"} Oct 31 03:00:36.099188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:36.099150 4456 generic.go:356] "Generic (PLEG): container finished" podID="96c66979-ebac-48ab-bc68-431eb8d495b1" containerID="33b442cb8a0d9fc6c11be5ebe38b94487bdaa526535470f5ed394f1d9bb0b5f7" exitCode=0 Oct 31 03:00:36.099969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:36.099943 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" event={"ID":"96c66979-ebac-48ab-bc68-431eb8d495b1","Type":"ContainerDied","Data":"33b442cb8a0d9fc6c11be5ebe38b94487bdaa526535470f5ed394f1d9bb0b5f7"} Oct 31 03:00:36.104901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:36.104654 4456 generic.go:356] "Generic (PLEG): container finished" podID="0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" containerID="bdb58614be8eafa52cf1f2808060ad1d4f280caee26c7fabd8926ecb775699c9" exitCode=0 Oct 31 03:00:36.104901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:36.104788 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" event={"ID":"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2","Type":"ContainerDied","Data":"bdb58614be8eafa52cf1f2808060ad1d4f280caee26c7fabd8926ecb775699c9"} Oct 31 03:00:36.121995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:36.121963 4456 generic.go:356] "Generic (PLEG): container finished" podID="13ae11bf-a60a-4f89-b078-ede5f7a80788" containerID="b3cdfe364ba2db3c07173b93f706fd7c3ddc348fdb5c750883a7cd9fbbf96abd" exitCode=0 Oct 31 03:00:36.122121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:36.122047 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" event={"ID":"13ae11bf-a60a-4f89-b078-ede5f7a80788","Type":"ContainerDied","Data":"b3cdfe364ba2db3c07173b93f706fd7c3ddc348fdb5c750883a7cd9fbbf96abd"} Oct 31 03:00:37.134866 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.134825 4456 generic.go:356] "Generic (PLEG): container finished" podID="b3e014e7-5988-443b-9543-45e65feef843" containerID="4e14d83003f48db6507f3f80aaee74442ba3aa11db016cc03ecaae8f5a04dd8a" exitCode=0 Oct 31 03:00:37.135352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.134944 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" event={"ID":"b3e014e7-5988-443b-9543-45e65feef843","Type":"ContainerDied","Data":"4e14d83003f48db6507f3f80aaee74442ba3aa11db016cc03ecaae8f5a04dd8a"} Oct 31 03:00:37.144727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.143834 4456 generic.go:356] "Generic (PLEG): container finished" podID="a7857fff-dff2-4b03-b39d-aec834cff058" containerID="7d644650b9d7c07f20dcea4ab7b6b99375a7f086fec09a7898f402b11ab82ff2" exitCode=0 Oct 31 03:00:37.144727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.143980 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" event={"ID":"a7857fff-dff2-4b03-b39d-aec834cff058","Type":"ContainerDied","Data":"7d644650b9d7c07f20dcea4ab7b6b99375a7f086fec09a7898f402b11ab82ff2"} Oct 31 03:00:37.148321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.148266 4456 generic.go:356] "Generic (PLEG): container finished" podID="6f7a4303-69ba-4f48-916c-1b9b99f477e6" containerID="279b76542d9733ff836e266f8331243197a906e00cd5c48e171a8983763259b0" exitCode=0 Oct 31 03:00:37.148482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.148357 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" event={"ID":"6f7a4303-69ba-4f48-916c-1b9b99f477e6","Type":"ContainerDied","Data":"279b76542d9733ff836e266f8331243197a906e00cd5c48e171a8983763259b0"} Oct 31 03:00:37.151759 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.151736 4456 generic.go:356] "Generic (PLEG): container finished" podID="b8039e8a-b518-4a48-94ce-a94333112d30" containerID="0abf962e633d8b1837767eaf2db84c290e2ec04cee141c75d09761ad7dc70661" exitCode=0 Oct 31 03:00:37.151873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.151844 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" event={"ID":"b8039e8a-b518-4a48-94ce-a94333112d30","Type":"ContainerDied","Data":"0abf962e633d8b1837767eaf2db84c290e2ec04cee141c75d09761ad7dc70661"} Oct 31 03:00:37.155739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.155700 4456 generic.go:356] "Generic (PLEG): container finished" podID="3657424b-e958-4a2a-aed9-90c659954a62" containerID="e8f73ab4b223c09c626f43b2a11ec254c8275200ff3635d9d230edc5796d7d33" exitCode=0 Oct 31 03:00:37.155834 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.155787 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" event={"ID":"3657424b-e958-4a2a-aed9-90c659954a62","Type":"ContainerDied","Data":"e8f73ab4b223c09c626f43b2a11ec254c8275200ff3635d9d230edc5796d7d33"} Oct 31 03:00:37.163417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.161523 4456 generic.go:356] "Generic (PLEG): container finished" podID="aa862741-8e25-486a-a803-e5c617f6dd0d" containerID="402c70560580d2a429198e7d3a20cfdb07431cb5714a3123f7fa7ac21eb228bc" exitCode=0 Oct 31 03:00:37.163417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.162963 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" event={"ID":"aa862741-8e25-486a-a803-e5c617f6dd0d","Type":"ContainerDied","Data":"402c70560580d2a429198e7d3a20cfdb07431cb5714a3123f7fa7ac21eb228bc"} Oct 31 03:00:37.172347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.172324 4456 generic.go:356] "Generic (PLEG): container finished" podID="ff332518-f459-4504-813f-b64fabf4001a" containerID="676f645f8790fccefb03db53747b17ee33b755804922a86efd0664fab348bf5d" exitCode=0 Oct 31 03:00:37.172420 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:37.172359 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" event={"ID":"ff332518-f459-4504-813f-b64fabf4001a","Type":"ContainerDied","Data":"676f645f8790fccefb03db53747b17ee33b755804922a86efd0664fab348bf5d"} Oct 31 03:00:40.251392 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:40.250123 4456 generic.go:356] "Generic (PLEG): container finished" podID="4a5143b6-190a-4cb9-83c8-41fdb4d1619a" containerID="2685ba30901e82dbe8a012d06b15cfca8df06580cfd23b84dd9064a4b3fb6fdb" exitCode=0 Oct 31 03:00:40.251392 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:40.250259 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" event={"ID":"4a5143b6-190a-4cb9-83c8-41fdb4d1619a","Type":"ContainerDied","Data":"2685ba30901e82dbe8a012d06b15cfca8df06580cfd23b84dd9064a4b3fb6fdb"} Oct 31 03:00:54.515464 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:54.514513 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" event={"ID":"b21ae90f-021c-45a3-ab03-7ec953df43ab","Type":"ContainerStarted","Data":"db9d580df605e489c30e6c35e7c5b03f4d54479e49d0cc7309954f3cf3242e4b"} Oct 31 03:00:54.539243 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:54.538969 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" podStartSLOduration=2.134968327 podStartE2EDuration="39.538956409s" totalImagesPullingTime="37.403988082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:00:54.537743759 +0000 UTC m=+336.218269890" watchObservedRunningTime="2026-10-31 03:00:54.538956409 +0000 UTC m=+336.219482535" Oct 31 03:00:55.540162 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:55.540127 4456 generic.go:356] "Generic (PLEG): container finished" podID="b21ae90f-021c-45a3-ab03-7ec953df43ab" containerID="db9d580df605e489c30e6c35e7c5b03f4d54479e49d0cc7309954f3cf3242e4b" exitCode=0 Oct 31 03:00:55.540684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:55.540176 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" event={"ID":"b21ae90f-021c-45a3-ab03-7ec953df43ab","Type":"ContainerDied","Data":"db9d580df605e489c30e6c35e7c5b03f4d54479e49d0cc7309954f3cf3242e4b"} Oct 31 03:00:56.554973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:56.554314 4456 generic.go:356] "Generic (PLEG): container finished" podID="4a5143b6-190a-4cb9-83c8-41fdb4d1619a" containerID="00896c5c655cc9f2af74e750e6a927892f6c4b076659d36edd113b9e4b731cc6" exitCode=0 Oct 31 03:00:56.554973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:56.554905 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" event={"ID":"4a5143b6-190a-4cb9-83c8-41fdb4d1619a","Type":"ContainerDied","Data":"00896c5c655cc9f2af74e750e6a927892f6c4b076659d36edd113b9e4b731cc6"} Oct 31 03:00:56.783342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:56.783311 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:56.922083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:56.921387 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-util\" (UniqueName: \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-util\") pod \"b21ae90f-021c-45a3-ab03-7ec953df43ab\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " Oct 31 03:00:56.922083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:56.921481 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21ae90f-021c-45a3-ab03-7ec953df43ab-kube-api-access-q6c44\" (UniqueName: \"kubernetes.io/projected/b21ae90f-021c-45a3-ab03-7ec953df43ab-kube-api-access-q6c44\") pod \"b21ae90f-021c-45a3-ab03-7ec953df43ab\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " Oct 31 03:00:56.922083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:56.921593 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-bundle\") pod \"b21ae90f-021c-45a3-ab03-7ec953df43ab\" (UID: \"b21ae90f-021c-45a3-ab03-7ec953df43ab\") " Oct 31 03:00:56.923322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:56.923274 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-bundle" pod "b21ae90f-021c-45a3-ab03-7ec953df43ab" (UID: "b21ae90f-021c-45a3-ab03-7ec953df43ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:00:56.926381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:56.926299 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21ae90f-021c-45a3-ab03-7ec953df43ab-kube-api-access-q6c44" pod "b21ae90f-021c-45a3-ab03-7ec953df43ab" (UID: "b21ae90f-021c-45a3-ab03-7ec953df43ab"). InnerVolumeSpecName "kube-api-access-q6c44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:00:56.929137 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:56.929106 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-util" pod "b21ae90f-021c-45a3-ab03-7ec953df43ab" (UID: "b21ae90f-021c-45a3-ab03-7ec953df43ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:00:57.023116 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.023075 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:00:57.023116 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.023121 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6c44\" (UniqueName: \"kubernetes.io/projected/b21ae90f-021c-45a3-ab03-7ec953df43ab-kube-api-access-q6c44\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:00:57.023390 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.023139 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b21ae90f-021c-45a3-ab03-7ec953df43ab-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:00:57.574065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.573379 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" Oct 31 03:00:57.574783 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.574574 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716c2f8" event={"ID":"b21ae90f-021c-45a3-ab03-7ec953df43ab","Type":"ContainerDied","Data":"aa8ff5b94782542ad6f3e8fd7fee22946bc56bb7a02681af9f9ac4462d1a71f6"} Oct 31 03:00:57.574783 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.574629 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8ff5b94782542ad6f3e8fd7fee22946bc56bb7a02681af9f9ac4462d1a71f6" Oct 31 03:00:57.843187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.842782 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:57.934550 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.933670 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-util\" (UniqueName: \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-util\") pod \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " Oct 31 03:00:57.934550 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.933737 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-kube-api-access-qxdkk\" (UniqueName: \"kubernetes.io/projected/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-kube-api-access-qxdkk\") pod \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " Oct 31 03:00:57.934550 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.933871 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-bundle\") pod \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\" (UID: \"4a5143b6-190a-4cb9-83c8-41fdb4d1619a\") " Oct 31 03:00:57.937935 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.937878 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-bundle" pod "4a5143b6-190a-4cb9-83c8-41fdb4d1619a" (UID: "4a5143b6-190a-4cb9-83c8-41fdb4d1619a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:00:57.948151 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.948030 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-util" pod "4a5143b6-190a-4cb9-83c8-41fdb4d1619a" (UID: "4a5143b6-190a-4cb9-83c8-41fdb4d1619a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:00:57.951919 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:57.951873 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-kube-api-access-qxdkk" pod "4a5143b6-190a-4cb9-83c8-41fdb4d1619a" (UID: "4a5143b6-190a-4cb9-83c8-41fdb4d1619a"). InnerVolumeSpecName "kube-api-access-qxdkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:00:58.035710 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:58.035674 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:00:58.035710 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:58.035707 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:00:58.035710 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:58.035717 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxdkk\" (UniqueName: \"kubernetes.io/projected/4a5143b6-190a-4cb9-83c8-41fdb4d1619a-kube-api-access-qxdkk\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:00:58.587082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:58.586186 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" event={"ID":"4a5143b6-190a-4cb9-83c8-41fdb4d1619a","Type":"ContainerDied","Data":"996eb78e77b2b076c28a8ea844c457efb7ba57a08c0c10038069fa3a4df9eff1"} Oct 31 03:00:58.587082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:58.586255 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="996eb78e77b2b076c28a8ea844c457efb7ba57a08c0c10038069fa3a4df9eff1" Oct 31 03:00:58.587082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:58.586343 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgp4d2" Oct 31 03:00:59.606794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.606711 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" event={"ID":"13ae11bf-a60a-4f89-b078-ede5f7a80788","Type":"ContainerStarted","Data":"870c9c9c748ed49dd9e6e67bfb3ea6ec3a061bb404338928d895d8fa8c269a4f"} Oct 31 03:00:59.612199 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.612144 4456 generic.go:356] "Generic (PLEG): container finished" podID="0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" containerID="8f65d24cdd57ba338c912c55c58b18c384edf36aec12c74d0659fda0a08c6042" exitCode=0 Oct 31 03:00:59.612309 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.612217 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" event={"ID":"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2","Type":"ContainerDied","Data":"8f65d24cdd57ba338c912c55c58b18c384edf36aec12c74d0659fda0a08c6042"} Oct 31 03:00:59.634987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.634931 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" podStartSLOduration=3.485601542 podStartE2EDuration="49.634918084s" totalImagesPullingTime="46.149316542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:00:59.632458263 +0000 UTC m=+341.312984426" watchObservedRunningTime="2026-10-31 03:00:59.634918084 +0000 UTC m=+341.315444211" Oct 31 03:00:59.672854 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:00:59.672794 4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ae11bf_a60a_4f89_b078_ede5f7a80788.slice/crio-870c9c9c748ed49dd9e6e67bfb3ea6ec3a061bb404338928d895d8fa8c269a4f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ae11bf_a60a_4f89_b078_ede5f7a80788.slice/crio-conmon-870c9c9c748ed49dd9e6e67bfb3ea6ec3a061bb404338928d895d8fa8c269a4f.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:00:59.838661 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.838625 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl"] Oct 31 03:00:59.839358 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839334 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b21ae90f-021c-45a3-ab03-7ec953df43ab" containerName="extract" Oct 31 03:00:59.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839355 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21ae90f-021c-45a3-ab03-7ec953df43ab" containerName="extract" Oct 31 03:00:59.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839395 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b21ae90f-021c-45a3-ab03-7ec953df43ab" containerName="util" Oct 31 03:00:59.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839405 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21ae90f-021c-45a3-ab03-7ec953df43ab" containerName="util" Oct 31 03:00:59.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839428 4456 cpu_manager.go:414] "containerMap: removing container" podUID="4a5143b6-190a-4cb9-83c8-41fdb4d1619a" containerName="util" Oct 31 03:00:59.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839437 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5143b6-190a-4cb9-83c8-41fdb4d1619a" containerName="util" Oct 31 03:00:59.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839455 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b21ae90f-021c-45a3-ab03-7ec953df43ab" containerName="pull" Oct 31 03:00:59.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839462 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21ae90f-021c-45a3-ab03-7ec953df43ab" containerName="pull" Oct 31 03:00:59.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839485 4456 cpu_manager.go:414] "containerMap: removing container" podUID="4a5143b6-190a-4cb9-83c8-41fdb4d1619a" containerName="extract" Oct 31 03:00:59.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839493 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5143b6-190a-4cb9-83c8-41fdb4d1619a" containerName="extract" Oct 31 03:00:59.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839505 4456 cpu_manager.go:414] "containerMap: removing container" podUID="4a5143b6-190a-4cb9-83c8-41fdb4d1619a" containerName="pull" Oct 31 03:00:59.840013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839513 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5143b6-190a-4cb9-83c8-41fdb4d1619a" containerName="pull" Oct 31 03:00:59.840013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839617 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5143b6-190a-4cb9-83c8-41fdb4d1619a" containerName="extract" Oct 31 03:00:59.840013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.839630 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21ae90f-021c-45a3-ab03-7ec953df43ab" containerName="extract" Oct 31 03:00:59.854547 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.854521 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:00:59.872825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.871600 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 03:00:59.872825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.871775 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-xc62g\"" Oct 31 03:00:59.872825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.871889 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 03:00:59.872825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.871938 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 03:00:59.872825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.871998 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 03:00:59.872825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.872109 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 03:00:59.946275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.946223 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl"] Oct 31 03:00:59.958940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.958896 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/573ec2a0-2d75-479d-a447-1804dedf242a-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:00:59.959133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.958966 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/573ec2a0-2d75-479d-a447-1804dedf242a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:00:59.959133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.959105 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/573ec2a0-2d75-479d-a447-1804dedf242a-tmp\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:00:59.959256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.959172 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/573ec2a0-2d75-479d-a447-1804dedf242a-metrics-cert\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:00:59.959256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.959209 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwdb\" (UniqueName: \"kubernetes.io/projected/573ec2a0-2d75-479d-a447-1804dedf242a-kube-api-access-xdwdb\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:00:59.959256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:00:59.959242 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/573ec2a0-2d75-479d-a447-1804dedf242a-manifests\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.060691 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.060654 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/573ec2a0-2d75-479d-a447-1804dedf242a-tmp\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.060849 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.060736 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/573ec2a0-2d75-479d-a447-1804dedf242a-metrics-cert\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.060849 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.060774 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwdb\" (UniqueName: \"kubernetes.io/projected/573ec2a0-2d75-479d-a447-1804dedf242a-kube-api-access-xdwdb\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.060849 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.060809 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/573ec2a0-2d75-479d-a447-1804dedf242a-manifests\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.060961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.060853 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/573ec2a0-2d75-479d-a447-1804dedf242a-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.060961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.060914 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/573ec2a0-2d75-479d-a447-1804dedf242a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.062429 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.062403 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/573ec2a0-2d75-479d-a447-1804dedf242a-manifests\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.077372 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.077348 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/573ec2a0-2d75-479d-a447-1804dedf242a-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.077533 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.077514 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwdb\" (UniqueName: \"kubernetes.io/projected/573ec2a0-2d75-479d-a447-1804dedf242a-kube-api-access-xdwdb\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.077625 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.077604 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/573ec2a0-2d75-479d-a447-1804dedf242a-tmp\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.080601 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.080576 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/573ec2a0-2d75-479d-a447-1804dedf242a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.084183 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.084134 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/573ec2a0-2d75-479d-a447-1804dedf242a-metrics-cert\") pod \"opendatahub-operator-controller-manager-57b855755-j7fhl\" (UID: \"573ec2a0-2d75-479d-a447-1804dedf242a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.191996 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.191964 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:00.560410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.559833 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl"] Oct 31 03:01:00.623114 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.622197 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" event={"ID":"aa862741-8e25-486a-a803-e5c617f6dd0d","Type":"ContainerStarted","Data":"10f13875636954e9e582e06cd7eb4c79fe25d2d6be7bab95f0197291f1932a13"} Oct 31 03:01:00.625080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.624982 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" event={"ID":"573ec2a0-2d75-479d-a447-1804dedf242a","Type":"ContainerStarted","Data":"674b38cae185ddecba0112d9b6f53e49876579f33ca417f0c17ff8304b28f425"} Oct 31 03:01:00.628559 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.628261 4456 generic.go:356] "Generic (PLEG): container finished" podID="13ae11bf-a60a-4f89-b078-ede5f7a80788" containerID="870c9c9c748ed49dd9e6e67bfb3ea6ec3a061bb404338928d895d8fa8c269a4f" exitCode=0 Oct 31 03:01:00.628559 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.628374 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" event={"ID":"13ae11bf-a60a-4f89-b078-ede5f7a80788","Type":"ContainerDied","Data":"870c9c9c748ed49dd9e6e67bfb3ea6ec3a061bb404338928d895d8fa8c269a4f"} Oct 31 03:01:00.645635 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.645601 4456 generic.go:356] "Generic (PLEG): container finished" podID="96c66979-ebac-48ab-bc68-431eb8d495b1" containerID="2a30aee1c0828265f3855c94fcd7f004b207d7c28fa3da7d8b1c2ac8ab61c3c4" exitCode=0 Oct 31 03:01:00.645931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.645824 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" event={"ID":"96c66979-ebac-48ab-bc68-431eb8d495b1","Type":"ContainerDied","Data":"2a30aee1c0828265f3855c94fcd7f004b207d7c28fa3da7d8b1c2ac8ab61c3c4"} Oct 31 03:01:00.664571 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.664513 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" podStartSLOduration=3.155071337 podStartE2EDuration="40.664497681s" totalImagesPullingTime="37.509426344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:00.662856875 +0000 UTC m=+342.343383003" watchObservedRunningTime="2026-10-31 03:01:00.664497681 +0000 UTC m=+342.345023807" Oct 31 03:01:00.830180 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.830129 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:01:00.970815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.970765 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-bundle\") pod \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " Oct 31 03:01:00.971121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.970853 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-kube-api-access-4mwch\" (UniqueName: \"kubernetes.io/projected/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-kube-api-access-4mwch\") pod \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " Oct 31 03:01:00.971121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.970907 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-util\" (UniqueName: \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-util\") pod \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\" (UID: \"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2\") " Oct 31 03:01:00.971792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.971757 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-bundle" pod "0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" (UID: "0dd99703-7989-4cd3-a014-d6c8f4ebc4e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:00.974443 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.974414 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-kube-api-access-4mwch" pod "0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" (UID: "0dd99703-7989-4cd3-a014-d6c8f4ebc4e2"). InnerVolumeSpecName "kube-api-access-4mwch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:01:00.978005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:00.977977 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-util" pod "0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" (UID: "0dd99703-7989-4cd3-a014-d6c8f4ebc4e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:01.072500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.072269 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mwch\" (UniqueName: \"kubernetes.io/projected/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-kube-api-access-4mwch\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:01.072500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.072312 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:01.072500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.072329 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dd99703-7989-4cd3-a014-d6c8f4ebc4e2-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:01.657587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.657549 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" Oct 31 03:01:01.658064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.657548 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jtdrc" event={"ID":"0dd99703-7989-4cd3-a014-d6c8f4ebc4e2","Type":"ContainerDied","Data":"fd9b1e87bf84261e12ed464142c8b10ed475a3ad724621815def0b3d9f98c6c6"} Oct 31 03:01:01.658064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.657687 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd9b1e87bf84261e12ed464142c8b10ed475a3ad724621815def0b3d9f98c6c6" Oct 31 03:01:01.665362 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.665323 4456 generic.go:356] "Generic (PLEG): container finished" podID="aa862741-8e25-486a-a803-e5c617f6dd0d" containerID="10f13875636954e9e582e06cd7eb4c79fe25d2d6be7bab95f0197291f1932a13" exitCode=0 Oct 31 03:01:01.665529 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.665433 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" event={"ID":"aa862741-8e25-486a-a803-e5c617f6dd0d","Type":"ContainerDied","Data":"10f13875636954e9e582e06cd7eb4c79fe25d2d6be7bab95f0197291f1932a13"} Oct 31 03:01:01.669168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.669125 4456 generic.go:356] "Generic (PLEG): container finished" podID="ff332518-f459-4504-813f-b64fabf4001a" containerID="91eb7c2433e3a4eb4c98e51b8e34c585382e5ce03ac9697e818e233fa174aafe" exitCode=0 Oct 31 03:01:01.669604 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.669578 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" event={"ID":"ff332518-f459-4504-813f-b64fabf4001a","Type":"ContainerDied","Data":"91eb7c2433e3a4eb4c98e51b8e34c585382e5ce03ac9697e818e233fa174aafe"} Oct 31 03:01:01.862125 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.862021 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:01:01.899835 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.899467 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:01:01.980258 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.980165 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ae11bf-a60a-4f89-b078-ede5f7a80788-kube-api-access-x47l8\" (UniqueName: \"kubernetes.io/projected/13ae11bf-a60a-4f89-b078-ede5f7a80788-kube-api-access-x47l8\") pod \"13ae11bf-a60a-4f89-b078-ede5f7a80788\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " Oct 31 03:01:01.980397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.980301 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-util\" (UniqueName: \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-util\") pod \"96c66979-ebac-48ab-bc68-431eb8d495b1\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " Oct 31 03:01:01.980397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.980364 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c66979-ebac-48ab-bc68-431eb8d495b1-kube-api-access-d86fz\" (UniqueName: \"kubernetes.io/projected/96c66979-ebac-48ab-bc68-431eb8d495b1-kube-api-access-d86fz\") pod \"96c66979-ebac-48ab-bc68-431eb8d495b1\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " Oct 31 03:01:01.980509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.980415 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-util\" (UniqueName: \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-util\") pod \"13ae11bf-a60a-4f89-b078-ede5f7a80788\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " Oct 31 03:01:01.980509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.980461 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-bundle\") pod \"13ae11bf-a60a-4f89-b078-ede5f7a80788\" (UID: \"13ae11bf-a60a-4f89-b078-ede5f7a80788\") " Oct 31 03:01:01.980509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.980490 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-bundle\") pod \"96c66979-ebac-48ab-bc68-431eb8d495b1\" (UID: \"96c66979-ebac-48ab-bc68-431eb8d495b1\") " Oct 31 03:01:01.981442 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.981410 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-bundle" pod "96c66979-ebac-48ab-bc68-431eb8d495b1" (UID: "96c66979-ebac-48ab-bc68-431eb8d495b1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:01.983386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.983347 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-bundle" pod "13ae11bf-a60a-4f89-b078-ede5f7a80788" (UID: "13ae11bf-a60a-4f89-b078-ede5f7a80788"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:01.988150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.985971 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ae11bf-a60a-4f89-b078-ede5f7a80788-kube-api-access-x47l8" pod "13ae11bf-a60a-4f89-b078-ede5f7a80788" (UID: "13ae11bf-a60a-4f89-b078-ede5f7a80788"). InnerVolumeSpecName "kube-api-access-x47l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:01:01.988350 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.988318 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c66979-ebac-48ab-bc68-431eb8d495b1-kube-api-access-d86fz" pod "96c66979-ebac-48ab-bc68-431eb8d495b1" (UID: "96c66979-ebac-48ab-bc68-431eb8d495b1"). InnerVolumeSpecName "kube-api-access-d86fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:01:01.990861 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.990833 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-util" pod "13ae11bf-a60a-4f89-b078-ede5f7a80788" (UID: "13ae11bf-a60a-4f89-b078-ede5f7a80788"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:01.992471 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:01.992440 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-util" pod "96c66979-ebac-48ab-bc68-431eb8d495b1" (UID: "96c66979-ebac-48ab-bc68-431eb8d495b1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:02.081562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.081534 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:02.081692 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.081566 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86fz\" (UniqueName: \"kubernetes.io/projected/96c66979-ebac-48ab-bc68-431eb8d495b1-kube-api-access-d86fz\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:02.081692 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.081582 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:02.081692 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.081599 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ae11bf-a60a-4f89-b078-ede5f7a80788-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:02.081692 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.081614 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96c66979-ebac-48ab-bc68-431eb8d495b1-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:02.081692 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.081631 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47l8\" (UniqueName: \"kubernetes.io/projected/13ae11bf-a60a-4f89-b078-ede5f7a80788-kube-api-access-x47l8\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:02.159027 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:01:02.158961 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:01:02.678733 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.678647 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" event={"ID":"13ae11bf-a60a-4f89-b078-ede5f7a80788","Type":"ContainerDied","Data":"ea0c77743943e4a8023ec51d8dc097d0afa63d1f8545f15f561fdd87aaabc7a7"} Oct 31 03:01:02.678733 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.678707 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea0c77743943e4a8023ec51d8dc097d0afa63d1f8545f15f561fdd87aaabc7a7" Oct 31 03:01:02.678733 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.678729 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449rnwt" Oct 31 03:01:02.682438 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.681424 4456 generic.go:356] "Generic (PLEG): container finished" podID="b3e014e7-5988-443b-9543-45e65feef843" containerID="418eca3546b6fe0adbed5d48e739bb629d8d1c8886ad084492accf0958a8a73f" exitCode=0 Oct 31 03:01:02.682556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.682511 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" event={"ID":"b3e014e7-5988-443b-9543-45e65feef843","Type":"ContainerDied","Data":"418eca3546b6fe0adbed5d48e739bb629d8d1c8886ad084492accf0958a8a73f"} Oct 31 03:01:02.685752 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.685725 4456 generic.go:356] "Generic (PLEG): container finished" podID="a7857fff-dff2-4b03-b39d-aec834cff058" containerID="7d0853aa99e60dd66897aaddbfadffc4add8b92b998e9b8e6380a1964851e443" exitCode=0 Oct 31 03:01:02.685860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.685766 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" event={"ID":"a7857fff-dff2-4b03-b39d-aec834cff058","Type":"ContainerDied","Data":"7d0853aa99e60dd66897aaddbfadffc4add8b92b998e9b8e6380a1964851e443"} Oct 31 03:01:02.690406 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.690381 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" Oct 31 03:01:02.690526 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.690419 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c48p9" event={"ID":"96c66979-ebac-48ab-bc68-431eb8d495b1","Type":"ContainerDied","Data":"c307da743cdc7a92b194ae45b24f76e41ff9773bef7993a22362ce5ac591ea62"} Oct 31 03:01:02.690526 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.690488 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c307da743cdc7a92b194ae45b24f76e41ff9773bef7993a22362ce5ac591ea62" Oct 31 03:01:02.693156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.693132 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" event={"ID":"6f7a4303-69ba-4f48-916c-1b9b99f477e6","Type":"ContainerStarted","Data":"f48a405a99e2b35d68729578d504b9e21bd1860695367a404a7c0f148e360f62"} Oct 31 03:01:02.695740 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.695717 4456 generic.go:356] "Generic (PLEG): container finished" podID="b8039e8a-b518-4a48-94ce-a94333112d30" containerID="48d4a16205ef407d95849ad63b542df0612fa927063b461448b0addbce4ade67" exitCode=0 Oct 31 03:01:02.695843 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.695793 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" event={"ID":"b8039e8a-b518-4a48-94ce-a94333112d30","Type":"ContainerDied","Data":"48d4a16205ef407d95849ad63b542df0612fa927063b461448b0addbce4ade67"} Oct 31 03:01:02.698348 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.698313 4456 generic.go:356] "Generic (PLEG): container finished" podID="3657424b-e958-4a2a-aed9-90c659954a62" containerID="3ee23d4323f8135b16d7943744d50cdba5d61ec0ae248ab16e240c63c6d7034a" exitCode=0 Oct 31 03:01:02.698450 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.698394 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" event={"ID":"3657424b-e958-4a2a-aed9-90c659954a62","Type":"ContainerDied","Data":"3ee23d4323f8135b16d7943744d50cdba5d61ec0ae248ab16e240c63c6d7034a"} Oct 31 03:01:02.848875 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.848848 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:01:02.883046 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.883009 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:01:02.992525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.992456 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa862741-8e25-486a-a803-e5c617f6dd0d-kube-api-access-mjbm2\" (UniqueName: \"kubernetes.io/projected/aa862741-8e25-486a-a803-e5c617f6dd0d-kube-api-access-mjbm2\") pod \"aa862741-8e25-486a-a803-e5c617f6dd0d\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " Oct 31 03:01:02.992525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.992519 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff332518-f459-4504-813f-b64fabf4001a-kube-api-access-tfff9\" (UniqueName: \"kubernetes.io/projected/ff332518-f459-4504-813f-b64fabf4001a-kube-api-access-tfff9\") pod \"ff332518-f459-4504-813f-b64fabf4001a\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " Oct 31 03:01:02.992730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.992549 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-util\" (UniqueName: \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-util\") pod \"aa862741-8e25-486a-a803-e5c617f6dd0d\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " Oct 31 03:01:02.992730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.992567 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-bundle\") pod \"ff332518-f459-4504-813f-b64fabf4001a\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " Oct 31 03:01:02.992730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.992622 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-util\" (UniqueName: \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-util\") pod \"ff332518-f459-4504-813f-b64fabf4001a\" (UID: \"ff332518-f459-4504-813f-b64fabf4001a\") " Oct 31 03:01:02.992730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.992654 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-bundle\") pod \"aa862741-8e25-486a-a803-e5c617f6dd0d\" (UID: \"aa862741-8e25-486a-a803-e5c617f6dd0d\") " Oct 31 03:01:02.993463 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.993436 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-bundle" pod "ff332518-f459-4504-813f-b64fabf4001a" (UID: "ff332518-f459-4504-813f-b64fabf4001a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:02.994620 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.994563 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-bundle" pod "aa862741-8e25-486a-a803-e5c617f6dd0d" (UID: "aa862741-8e25-486a-a803-e5c617f6dd0d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:02.994990 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.994959 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa862741-8e25-486a-a803-e5c617f6dd0d-kube-api-access-mjbm2" pod "aa862741-8e25-486a-a803-e5c617f6dd0d" (UID: "aa862741-8e25-486a-a803-e5c617f6dd0d"). InnerVolumeSpecName "kube-api-access-mjbm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:01:02.995096 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.995070 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff332518-f459-4504-813f-b64fabf4001a-kube-api-access-tfff9" pod "ff332518-f459-4504-813f-b64fabf4001a" (UID: "ff332518-f459-4504-813f-b64fabf4001a"). InnerVolumeSpecName "kube-api-access-tfff9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:01:02.998161 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:02.998138 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-util" pod "aa862741-8e25-486a-a803-e5c617f6dd0d" (UID: "aa862741-8e25-486a-a803-e5c617f6dd0d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:03.000473 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.000445 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-util" pod "ff332518-f459-4504-813f-b64fabf4001a" (UID: "ff332518-f459-4504-813f-b64fabf4001a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:03.094382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.094341 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfff9\" (UniqueName: \"kubernetes.io/projected/ff332518-f459-4504-813f-b64fabf4001a-kube-api-access-tfff9\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:03.094382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.094381 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:03.094581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.094395 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:03.094581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.094406 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff332518-f459-4504-813f-b64fabf4001a-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:03.094581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.094418 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa862741-8e25-486a-a803-e5c617f6dd0d-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:03.094581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.094426 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjbm2\" (UniqueName: \"kubernetes.io/projected/aa862741-8e25-486a-a803-e5c617f6dd0d-kube-api-access-mjbm2\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:03.706274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.706234 4456 generic.go:356] "Generic (PLEG): container finished" podID="6f7a4303-69ba-4f48-916c-1b9b99f477e6" containerID="f48a405a99e2b35d68729578d504b9e21bd1860695367a404a7c0f148e360f62" exitCode=0 Oct 31 03:01:03.706731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.706331 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" event={"ID":"6f7a4303-69ba-4f48-916c-1b9b99f477e6","Type":"ContainerDied","Data":"f48a405a99e2b35d68729578d504b9e21bd1860695367a404a7c0f148e360f62"} Oct 31 03:01:03.708640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.708609 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" event={"ID":"aa862741-8e25-486a-a803-e5c617f6dd0d","Type":"ContainerDied","Data":"d04bcbc002b545e5a70d00fb57e1e3a701b3c9e4f6e4e0eef53f6251b2a582e0"} Oct 31 03:01:03.709250 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.709220 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcv9l" Oct 31 03:01:03.710265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.708661 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d04bcbc002b545e5a70d00fb57e1e3a701b3c9e4f6e4e0eef53f6251b2a582e0" Oct 31 03:01:03.720057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.713401 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" event={"ID":"ff332518-f459-4504-813f-b64fabf4001a","Type":"ContainerDied","Data":"d13c0dfb61ae8198d535f5d4d5cb015975a4a1c9741bd44b97ecc9f3a19046a8"} Oct 31 03:01:03.720057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.713471 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d13c0dfb61ae8198d535f5d4d5cb015975a4a1c9741bd44b97ecc9f3a19046a8" Oct 31 03:01:03.720057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.713574 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6xd2d" Oct 31 03:01:03.917745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.917713 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:01:03.950513 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.950483 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:01:03.962831 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.962766 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:01:03.965392 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.965367 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:01:03.988668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:03.988639 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:01:04.002830 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.002800 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-bundle\") pod \"3657424b-e958-4a2a-aed9-90c659954a62\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " Oct 31 03:01:04.003148 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.002869 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3657424b-e958-4a2a-aed9-90c659954a62-kube-api-access-gz4ld\" (UniqueName: \"kubernetes.io/projected/3657424b-e958-4a2a-aed9-90c659954a62-kube-api-access-gz4ld\") pod \"3657424b-e958-4a2a-aed9-90c659954a62\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " Oct 31 03:01:04.003148 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.002935 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-util\" (UniqueName: \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-util\") pod \"3657424b-e958-4a2a-aed9-90c659954a62\" (UID: \"3657424b-e958-4a2a-aed9-90c659954a62\") " Oct 31 03:01:04.003759 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.003731 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-bundle" pod "3657424b-e958-4a2a-aed9-90c659954a62" (UID: "3657424b-e958-4a2a-aed9-90c659954a62"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:04.009370 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.009341 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3657424b-e958-4a2a-aed9-90c659954a62-kube-api-access-gz4ld" pod "3657424b-e958-4a2a-aed9-90c659954a62" (UID: "3657424b-e958-4a2a-aed9-90c659954a62"). InnerVolumeSpecName "kube-api-access-gz4ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:01:04.009975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.009944 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-util" pod "3657424b-e958-4a2a-aed9-90c659954a62" (UID: "3657424b-e958-4a2a-aed9-90c659954a62"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:04.104154 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104118 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-util\" (UniqueName: \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-util\") pod \"b8039e8a-b518-4a48-94ce-a94333112d30\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " Oct 31 03:01:04.104334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104183 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8039e8a-b518-4a48-94ce-a94333112d30-kube-api-access-t4pzq\" (UniqueName: \"kubernetes.io/projected/b8039e8a-b518-4a48-94ce-a94333112d30-kube-api-access-t4pzq\") pod \"b8039e8a-b518-4a48-94ce-a94333112d30\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " Oct 31 03:01:04.104334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104205 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-util\" (UniqueName: \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-util\") pod \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " Oct 31 03:01:04.104334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104235 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-bundle\") pod \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " Oct 31 03:01:04.104334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104265 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7857fff-dff2-4b03-b39d-aec834cff058-kube-api-access-xpzgg\" (UniqueName: \"kubernetes.io/projected/a7857fff-dff2-4b03-b39d-aec834cff058-kube-api-access-xpzgg\") pod \"a7857fff-dff2-4b03-b39d-aec834cff058\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " Oct 31 03:01:04.104334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104289 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-bundle\") pod \"a7857fff-dff2-4b03-b39d-aec834cff058\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " Oct 31 03:01:04.104334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104317 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-bundle\") pod \"b8039e8a-b518-4a48-94ce-a94333112d30\" (UID: \"b8039e8a-b518-4a48-94ce-a94333112d30\") " Oct 31 03:01:04.104623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104354 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-bundle\") pod \"b3e014e7-5988-443b-9543-45e65feef843\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " Oct 31 03:01:04.104623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104384 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7a4303-69ba-4f48-916c-1b9b99f477e6-kube-api-access-rmdpj\" (UniqueName: \"kubernetes.io/projected/6f7a4303-69ba-4f48-916c-1b9b99f477e6-kube-api-access-rmdpj\") pod \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\" (UID: \"6f7a4303-69ba-4f48-916c-1b9b99f477e6\") " Oct 31 03:01:04.104623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104436 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-util\" (UniqueName: \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-util\") pod \"b3e014e7-5988-443b-9543-45e65feef843\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " Oct 31 03:01:04.104623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104464 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e014e7-5988-443b-9543-45e65feef843-kube-api-access-9pcxn\" (UniqueName: \"kubernetes.io/projected/b3e014e7-5988-443b-9543-45e65feef843-kube-api-access-9pcxn\") pod \"b3e014e7-5988-443b-9543-45e65feef843\" (UID: \"b3e014e7-5988-443b-9543-45e65feef843\") " Oct 31 03:01:04.104623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104530 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-util\" (UniqueName: \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-util\") pod \"a7857fff-dff2-4b03-b39d-aec834cff058\" (UID: \"a7857fff-dff2-4b03-b39d-aec834cff058\") " Oct 31 03:01:04.104980 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104802 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.104980 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104824 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz4ld\" (UniqueName: \"kubernetes.io/projected/3657424b-e958-4a2a-aed9-90c659954a62-kube-api-access-gz4ld\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.104980 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.104840 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3657424b-e958-4a2a-aed9-90c659954a62-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.105424 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.105335 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-bundle" pod "a7857fff-dff2-4b03-b39d-aec834cff058" (UID: "a7857fff-dff2-4b03-b39d-aec834cff058"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:04.105619 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.105487 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-bundle" pod "6f7a4303-69ba-4f48-916c-1b9b99f477e6" (UID: "6f7a4303-69ba-4f48-916c-1b9b99f477e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:04.106260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.106178 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-bundle" pod "b8039e8a-b518-4a48-94ce-a94333112d30" (UID: "b8039e8a-b518-4a48-94ce-a94333112d30"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:04.110169 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.110090 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-bundle" pod "b3e014e7-5988-443b-9543-45e65feef843" (UID: "b3e014e7-5988-443b-9543-45e65feef843"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:04.110812 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.110329 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7a4303-69ba-4f48-916c-1b9b99f477e6-kube-api-access-rmdpj" pod "6f7a4303-69ba-4f48-916c-1b9b99f477e6" (UID: "6f7a4303-69ba-4f48-916c-1b9b99f477e6"). InnerVolumeSpecName "kube-api-access-rmdpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:01:04.114401 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.114359 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-util" pod "6f7a4303-69ba-4f48-916c-1b9b99f477e6" (UID: "6f7a4303-69ba-4f48-916c-1b9b99f477e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:04.114925 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.114881 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-util" pod "b3e014e7-5988-443b-9543-45e65feef843" (UID: "b3e014e7-5988-443b-9543-45e65feef843"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:04.115031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.115002 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-util" pod "a7857fff-dff2-4b03-b39d-aec834cff058" (UID: "a7857fff-dff2-4b03-b39d-aec834cff058"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:04.115797 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.115640 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8039e8a-b518-4a48-94ce-a94333112d30-kube-api-access-t4pzq" pod "b8039e8a-b518-4a48-94ce-a94333112d30" (UID: "b8039e8a-b518-4a48-94ce-a94333112d30"). InnerVolumeSpecName "kube-api-access-t4pzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:01:04.115797 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.115769 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7857fff-dff2-4b03-b39d-aec834cff058-kube-api-access-xpzgg" pod "a7857fff-dff2-4b03-b39d-aec834cff058" (UID: "a7857fff-dff2-4b03-b39d-aec834cff058"). InnerVolumeSpecName "kube-api-access-xpzgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:01:04.117281 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.117245 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e014e7-5988-443b-9543-45e65feef843-kube-api-access-9pcxn" pod "b3e014e7-5988-443b-9543-45e65feef843" (UID: "b3e014e7-5988-443b-9543-45e65feef843"). InnerVolumeSpecName "kube-api-access-9pcxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:01:04.123543 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.123516 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-util" pod "b8039e8a-b518-4a48-94ce-a94333112d30" (UID: "b8039e8a-b518-4a48-94ce-a94333112d30"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:01:04.205442 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205403 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205442 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205432 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmdpj\" (UniqueName: \"kubernetes.io/projected/6f7a4303-69ba-4f48-916c-1b9b99f477e6-kube-api-access-rmdpj\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205442 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205441 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3e014e7-5988-443b-9543-45e65feef843-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205442 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205450 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pcxn\" (UniqueName: \"kubernetes.io/projected/b3e014e7-5988-443b-9543-45e65feef843-kube-api-access-9pcxn\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205459 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205468 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205476 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4pzq\" (UniqueName: \"kubernetes.io/projected/b8039e8a-b518-4a48-94ce-a94333112d30-kube-api-access-t4pzq\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205489 4456 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-util\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205502 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7a4303-69ba-4f48-916c-1b9b99f477e6-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205516 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpzgg\" (UniqueName: \"kubernetes.io/projected/a7857fff-dff2-4b03-b39d-aec834cff058-kube-api-access-xpzgg\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205530 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7857fff-dff2-4b03-b39d-aec834cff058-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.205721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.205543 4456 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8039e8a-b518-4a48-94ce-a94333112d30-bundle\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:01:04.720115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.720019 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" Oct 31 03:01:04.720115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.720059 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxtdww" event={"ID":"b3e014e7-5988-443b-9543-45e65feef843","Type":"ContainerDied","Data":"853d5c91a54474ade7316cead8b9cde6c002803a224fd093bc83ca4b28ccf470"} Oct 31 03:01:04.720573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.720116 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="853d5c91a54474ade7316cead8b9cde6c002803a224fd093bc83ca4b28ccf470" Oct 31 03:01:04.722224 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.722202 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" Oct 31 03:01:04.722357 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.722223 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78gbgw" event={"ID":"a7857fff-dff2-4b03-b39d-aec834cff058","Type":"ContainerDied","Data":"fd4eaec01de32e93a8a2b0735719f9abd1cafc3d1a51922372ef656b1c289764"} Oct 31 03:01:04.722357 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.722268 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd4eaec01de32e93a8a2b0735719f9abd1cafc3d1a51922372ef656b1c289764" Oct 31 03:01:04.724375 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.724355 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" event={"ID":"6f7a4303-69ba-4f48-916c-1b9b99f477e6","Type":"ContainerDied","Data":"bcc7cf49a2e2b429ec5e3f4ff2c63ce98f61732d657e897e9dac48670835a3df"} Oct 31 03:01:04.724473 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.724390 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcc7cf49a2e2b429ec5e3f4ff2c63ce98f61732d657e897e9dac48670835a3df" Oct 31 03:01:04.724473 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.724444 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jjdg6" Oct 31 03:01:04.726855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.726841 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" Oct 31 03:01:04.726855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.726846 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwm6vk" event={"ID":"b8039e8a-b518-4a48-94ce-a94333112d30","Type":"ContainerDied","Data":"c26101f80fa3006e9a1800a4c607dc21e28e81f2bcca343878dad18f73a11d3e"} Oct 31 03:01:04.726969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.726873 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26101f80fa3006e9a1800a4c607dc21e28e81f2bcca343878dad18f73a11d3e" Oct 31 03:01:04.729019 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.729003 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" Oct 31 03:01:04.729126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.729031 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arx6vb" event={"ID":"3657424b-e958-4a2a-aed9-90c659954a62","Type":"ContainerDied","Data":"dcb5a24a164f88e5b28b4878f95f85dafafa3a80396dd40e872f6666b48e36ac"} Oct 31 03:01:04.729126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.729091 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcb5a24a164f88e5b28b4878f95f85dafafa3a80396dd40e872f6666b48e36ac" Oct 31 03:01:04.730632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:04.730606 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" event={"ID":"573ec2a0-2d75-479d-a447-1804dedf242a","Type":"ContainerStarted","Data":"1de6b751ab9515645d8431f8830c7be4a4f541018e38f1f407300a8eed15fc7c"} Oct 31 03:01:05.737661 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:05.737625 4456 generic.go:356] "Generic (PLEG): container finished" podID="573ec2a0-2d75-479d-a447-1804dedf242a" containerID="1de6b751ab9515645d8431f8830c7be4a4f541018e38f1f407300a8eed15fc7c" exitCode=0 Oct 31 03:01:05.738096 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:05.737698 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" event={"ID":"573ec2a0-2d75-479d-a447-1804dedf242a","Type":"ContainerDied","Data":"1de6b751ab9515645d8431f8830c7be4a4f541018e38f1f407300a8eed15fc7c"} Oct 31 03:01:06.745468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:06.745427 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" event={"ID":"573ec2a0-2d75-479d-a447-1804dedf242a","Type":"ContainerStarted","Data":"eaf369982a7ec8a554507ac3e07fd1422c7a03001430b5211a9a9159acd214bd"} Oct 31 03:01:06.745862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:06.745598 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:06.797767 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:06.797709 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" podStartSLOduration=2.978395701 podStartE2EDuration="7.797694992s" totalImagesPullingTime="4.390502426s" totalInitContainerRuntime="428.796865ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:06.793428815 +0000 UTC m=+348.473954934" watchObservedRunningTime="2026-10-31 03:01:06.797694992 +0000 UTC m=+348.478221172" Oct 31 03:01:08.275600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275568 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs"] Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275898 4456 cpu_manager.go:414] "containerMap: removing container" podUID="0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" containerName="pull" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275909 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" containerName="pull" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275918 4456 cpu_manager.go:414] "containerMap: removing container" podUID="ff332518-f459-4504-813f-b64fabf4001a" containerName="pull" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275923 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff332518-f459-4504-813f-b64fabf4001a" containerName="pull" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275930 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b3e014e7-5988-443b-9543-45e65feef843" containerName="extract" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275935 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e014e7-5988-443b-9543-45e65feef843" containerName="extract" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275944 4456 cpu_manager.go:414] "containerMap: removing container" podUID="3657424b-e958-4a2a-aed9-90c659954a62" containerName="util" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275949 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3657424b-e958-4a2a-aed9-90c659954a62" containerName="util" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275954 4456 cpu_manager.go:414] "containerMap: removing container" podUID="aa862741-8e25-486a-a803-e5c617f6dd0d" containerName="extract" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275959 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa862741-8e25-486a-a803-e5c617f6dd0d" containerName="extract" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275966 4456 cpu_manager.go:414] "containerMap: removing container" podUID="6f7a4303-69ba-4f48-916c-1b9b99f477e6" containerName="extract" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275971 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7a4303-69ba-4f48-916c-1b9b99f477e6" containerName="extract" Oct 31 03:01:08.275973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275979 4456 cpu_manager.go:414] "containerMap: removing container" podUID="0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275983 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275990 4456 cpu_manager.go:414] "containerMap: removing container" podUID="13ae11bf-a60a-4f89-b078-ede5f7a80788" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.275994 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ae11bf-a60a-4f89-b078-ede5f7a80788" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276000 4456 cpu_manager.go:414] "containerMap: removing container" podUID="6f7a4303-69ba-4f48-916c-1b9b99f477e6" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276004 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7a4303-69ba-4f48-916c-1b9b99f477e6" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276011 4456 cpu_manager.go:414] "containerMap: removing container" podUID="aa862741-8e25-486a-a803-e5c617f6dd0d" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276016 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa862741-8e25-486a-a803-e5c617f6dd0d" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276020 4456 cpu_manager.go:414] "containerMap: removing container" podUID="a7857fff-dff2-4b03-b39d-aec834cff058" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276024 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7857fff-dff2-4b03-b39d-aec834cff058" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276053 4456 cpu_manager.go:414] "containerMap: removing container" podUID="96c66979-ebac-48ab-bc68-431eb8d495b1" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276059 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c66979-ebac-48ab-bc68-431eb8d495b1" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276069 4456 cpu_manager.go:414] "containerMap: removing container" podUID="a7857fff-dff2-4b03-b39d-aec834cff058" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276075 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7857fff-dff2-4b03-b39d-aec834cff058" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276081 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b8039e8a-b518-4a48-94ce-a94333112d30" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276085 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8039e8a-b518-4a48-94ce-a94333112d30" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276091 4456 cpu_manager.go:414] "containerMap: removing container" podUID="aa862741-8e25-486a-a803-e5c617f6dd0d" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276095 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa862741-8e25-486a-a803-e5c617f6dd0d" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276101 4456 cpu_manager.go:414] "containerMap: removing container" podUID="13ae11bf-a60a-4f89-b078-ede5f7a80788" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276105 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ae11bf-a60a-4f89-b078-ede5f7a80788" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276112 4456 cpu_manager.go:414] "containerMap: removing container" podUID="96c66979-ebac-48ab-bc68-431eb8d495b1" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276116 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c66979-ebac-48ab-bc68-431eb8d495b1" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276122 4456 cpu_manager.go:414] "containerMap: removing container" podUID="0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276128 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276134 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b8039e8a-b518-4a48-94ce-a94333112d30" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276138 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8039e8a-b518-4a48-94ce-a94333112d30" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276143 4456 cpu_manager.go:414] "containerMap: removing container" podUID="3657424b-e958-4a2a-aed9-90c659954a62" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276148 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3657424b-e958-4a2a-aed9-90c659954a62" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276155 4456 cpu_manager.go:414] "containerMap: removing container" podUID="ff332518-f459-4504-813f-b64fabf4001a" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276160 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff332518-f459-4504-813f-b64fabf4001a" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276166 4456 cpu_manager.go:414] "containerMap: removing container" podUID="96c66979-ebac-48ab-bc68-431eb8d495b1" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276170 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c66979-ebac-48ab-bc68-431eb8d495b1" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276176 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b3e014e7-5988-443b-9543-45e65feef843" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276180 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e014e7-5988-443b-9543-45e65feef843" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276187 4456 cpu_manager.go:414] "containerMap: removing container" podUID="13ae11bf-a60a-4f89-b078-ede5f7a80788" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276192 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ae11bf-a60a-4f89-b078-ede5f7a80788" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276201 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b8039e8a-b518-4a48-94ce-a94333112d30" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276205 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8039e8a-b518-4a48-94ce-a94333112d30" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276215 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b3e014e7-5988-443b-9543-45e65feef843" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276219 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e014e7-5988-443b-9543-45e65feef843" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276226 4456 cpu_manager.go:414] "containerMap: removing container" podUID="6f7a4303-69ba-4f48-916c-1b9b99f477e6" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276230 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7a4303-69ba-4f48-916c-1b9b99f477e6" containerName="util" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276235 4456 cpu_manager.go:414] "containerMap: removing container" podUID="3657424b-e958-4a2a-aed9-90c659954a62" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276239 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3657424b-e958-4a2a-aed9-90c659954a62" containerName="pull" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276246 4456 cpu_manager.go:414] "containerMap: removing container" podUID="ff332518-f459-4504-813f-b64fabf4001a" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276252 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff332518-f459-4504-813f-b64fabf4001a" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276258 4456 cpu_manager.go:414] "containerMap: removing container" podUID="a7857fff-dff2-4b03-b39d-aec834cff058" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276262 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7857fff-dff2-4b03-b39d-aec834cff058" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276315 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c66979-ebac-48ab-bc68-431eb8d495b1" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276322 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ae11bf-a60a-4f89-b078-ede5f7a80788" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276328 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd99703-7989-4cd3-a014-d6c8f4ebc4e2" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276333 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8039e8a-b518-4a48-94ce-a94333112d30" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276338 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3657424b-e958-4a2a-aed9-90c659954a62" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276346 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7857fff-dff2-4b03-b39d-aec834cff058" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276351 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff332518-f459-4504-813f-b64fabf4001a" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276357 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e014e7-5988-443b-9543-45e65feef843" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276363 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa862741-8e25-486a-a803-e5c617f6dd0d" containerName="extract" Oct 31 03:01:08.276353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.276367 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7a4303-69ba-4f48-916c-1b9b99f477e6" containerName="extract" Oct 31 03:01:08.279481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.279465 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" Oct 31 03:01:08.292688 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.292655 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 03:01:08.292688 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.292679 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:01:08.295159 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.295142 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qjhfx\"" Oct 31 03:01:08.297399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.297380 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs"] Oct 31 03:01:08.344853 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.344818 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmg7m\" (UniqueName: \"kubernetes.io/projected/ccd3596a-7171-49f3-b297-eaadfdc2d424-kube-api-access-bmg7m\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs\" (UID: \"ccd3596a-7171-49f3-b297-eaadfdc2d424\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" Oct 31 03:01:08.345013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.344946 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccd3596a-7171-49f3-b297-eaadfdc2d424-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs\" (UID: \"ccd3596a-7171-49f3-b297-eaadfdc2d424\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" Oct 31 03:01:08.446093 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.446063 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccd3596a-7171-49f3-b297-eaadfdc2d424-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs\" (UID: \"ccd3596a-7171-49f3-b297-eaadfdc2d424\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" Oct 31 03:01:08.446260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.446112 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmg7m\" (UniqueName: \"kubernetes.io/projected/ccd3596a-7171-49f3-b297-eaadfdc2d424-kube-api-access-bmg7m\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs\" (UID: \"ccd3596a-7171-49f3-b297-eaadfdc2d424\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" Oct 31 03:01:08.446408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.446389 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccd3596a-7171-49f3-b297-eaadfdc2d424-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs\" (UID: \"ccd3596a-7171-49f3-b297-eaadfdc2d424\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" Oct 31 03:01:08.463578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.463549 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmg7m\" (UniqueName: \"kubernetes.io/projected/ccd3596a-7171-49f3-b297-eaadfdc2d424-kube-api-access-bmg7m\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs\" (UID: \"ccd3596a-7171-49f3-b297-eaadfdc2d424\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" Oct 31 03:01:08.590977 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.590895 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" Oct 31 03:01:08.873748 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:08.873712 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs"] Oct 31 03:01:08.874931 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:01:08.874903 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd3596a_7171_49f3_b297_eaadfdc2d424.slice/crio-f4d83ff061082cff9b5f3333fd54c09c80f47d90184f2cbb1db2d6b46df85027 WatchSource:0}: Error finding container f4d83ff061082cff9b5f3333fd54c09c80f47d90184f2cbb1db2d6b46df85027: Status 404 returned error can't find the container with id f4d83ff061082cff9b5f3333fd54c09c80f47d90184f2cbb1db2d6b46df85027 Oct 31 03:01:09.762014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:09.761979 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" event={"ID":"ccd3596a-7171-49f3-b297-eaadfdc2d424","Type":"ContainerStarted","Data":"f4d83ff061082cff9b5f3333fd54c09c80f47d90184f2cbb1db2d6b46df85027"} Oct 31 03:01:11.774642 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:11.774550 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" event={"ID":"ccd3596a-7171-49f3-b297-eaadfdc2d424","Type":"ContainerStarted","Data":"abc342af03af6005ceaf182db8d5e3be9539eab905ed3ce1dd5761861437ccf5"} Oct 31 03:01:11.803579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:11.803527 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xmrgs" podStartSLOduration=1.27048128 podStartE2EDuration="3.803516385s" totalImagesPullingTime="2.533035105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:11.801244071 +0000 UTC m=+353.481770200" watchObservedRunningTime="2026-10-31 03:01:11.803516385 +0000 UTC m=+353.484042511" Oct 31 03:01:12.780621 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.780593 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5"] Oct 31 03:01:12.784228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.784208 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" Oct 31 03:01:12.793986 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.793964 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:01:12.794160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.794024 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-hq4gw\"" Oct 31 03:01:12.794160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.794065 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 03:01:12.798207 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.798182 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5"] Oct 31 03:01:12.881357 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.881316 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcps9\" (UniqueName: \"kubernetes.io/projected/0ca82a38-f332-40e2-a44b-5a5cca7aaf4d-kube-api-access-kcps9\") pod \"jobset-operator-674cd59dc5-l9bq5\" (UID: \"0ca82a38-f332-40e2-a44b-5a5cca7aaf4d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" Oct 31 03:01:12.881515 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.881369 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ca82a38-f332-40e2-a44b-5a5cca7aaf4d-tmp\") pod \"jobset-operator-674cd59dc5-l9bq5\" (UID: \"0ca82a38-f332-40e2-a44b-5a5cca7aaf4d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" Oct 31 03:01:12.982786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.982747 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcps9\" (UniqueName: \"kubernetes.io/projected/0ca82a38-f332-40e2-a44b-5a5cca7aaf4d-kube-api-access-kcps9\") pod \"jobset-operator-674cd59dc5-l9bq5\" (UID: \"0ca82a38-f332-40e2-a44b-5a5cca7aaf4d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" Oct 31 03:01:12.982786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.982792 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ca82a38-f332-40e2-a44b-5a5cca7aaf4d-tmp\") pod \"jobset-operator-674cd59dc5-l9bq5\" (UID: \"0ca82a38-f332-40e2-a44b-5a5cca7aaf4d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" Oct 31 03:01:12.983195 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.983176 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ca82a38-f332-40e2-a44b-5a5cca7aaf4d-tmp\") pod \"jobset-operator-674cd59dc5-l9bq5\" (UID: \"0ca82a38-f332-40e2-a44b-5a5cca7aaf4d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" Oct 31 03:01:12.996647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:12.996622 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcps9\" (UniqueName: \"kubernetes.io/projected/0ca82a38-f332-40e2-a44b-5a5cca7aaf4d-kube-api-access-kcps9\") pod \"jobset-operator-674cd59dc5-l9bq5\" (UID: \"0ca82a38-f332-40e2-a44b-5a5cca7aaf4d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" Oct 31 03:01:13.108729 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.108637 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" Oct 31 03:01:13.403463 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.403074 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5"] Oct 31 03:01:13.409997 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:01:13.409966 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ca82a38_f332_40e2_a44b_5a5cca7aaf4d.slice/crio-ce57890dcf38f4311f170b4be677c0b50847c57b634014229063281babd4e147 WatchSource:0}: Error finding container ce57890dcf38f4311f170b4be677c0b50847c57b634014229063281babd4e147: Status 404 returned error can't find the container with id ce57890dcf38f4311f170b4be677c0b50847c57b634014229063281babd4e147 Oct 31 03:01:13.556103 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.556069 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q"] Oct 31 03:01:13.561204 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.561187 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" Oct 31 03:01:13.572516 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.572493 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nz5m4\"" Oct 31 03:01:13.572620 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.572557 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 03:01:13.572897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.572880 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 03:01:13.572994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.572979 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q"] Oct 31 03:01:13.688395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.688359 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq82l\" (UniqueName: \"kubernetes.io/projected/5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7-kube-api-access-wq82l\") pod \"cert-manager-webhook-5dbbbf75cc-hk27q\" (UID: \"5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" Oct 31 03:01:13.688573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.688437 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hk27q\" (UID: \"5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" Oct 31 03:01:13.786342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.786310 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" event={"ID":"0ca82a38-f332-40e2-a44b-5a5cca7aaf4d","Type":"ContainerStarted","Data":"ce57890dcf38f4311f170b4be677c0b50847c57b634014229063281babd4e147"} Oct 31 03:01:13.789696 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.789673 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq82l\" (UniqueName: \"kubernetes.io/projected/5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7-kube-api-access-wq82l\") pod \"cert-manager-webhook-5dbbbf75cc-hk27q\" (UID: \"5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" Oct 31 03:01:13.789765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.789734 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hk27q\" (UID: \"5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" Oct 31 03:01:13.803051 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.803013 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hk27q\" (UID: \"5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" Oct 31 03:01:13.803226 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.803207 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq82l\" (UniqueName: \"kubernetes.io/projected/5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7-kube-api-access-wq82l\") pod \"cert-manager-webhook-5dbbbf75cc-hk27q\" (UID: \"5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" Oct 31 03:01:13.873260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:13.873232 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" Oct 31 03:01:14.144934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:14.144902 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q"] Oct 31 03:01:14.146410 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:01:14.146383 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b013bce_d4ab_48d5_8e6e_fa2bb6d80aa7.slice/crio-2db48039ad6566c691d7bafe4cc3574ddbbd667f7767d5b6b9fe0cfaa96cb06a WatchSource:0}: Error finding container 2db48039ad6566c691d7bafe4cc3574ddbbd667f7767d5b6b9fe0cfaa96cb06a: Status 404 returned error can't find the container with id 2db48039ad6566c691d7bafe4cc3574ddbbd667f7767d5b6b9fe0cfaa96cb06a Oct 31 03:01:14.794368 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:14.794324 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" event={"ID":"5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7","Type":"ContainerStarted","Data":"2db48039ad6566c691d7bafe4cc3574ddbbd667f7767d5b6b9fe0cfaa96cb06a"} Oct 31 03:01:15.802983 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:15.802940 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" event={"ID":"0ca82a38-f332-40e2-a44b-5a5cca7aaf4d","Type":"ContainerStarted","Data":"201c7fd7d2ff2d8c1bc84b3ebed105d4dbf471231abb7791f622c309d1e81ae1"} Oct 31 03:01:15.825130 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:15.825079 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l9bq5" podStartSLOduration=1.578526787 podStartE2EDuration="3.825065599s" totalImagesPullingTime="2.246538812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:15.824353237 +0000 UTC m=+357.504879426" watchObservedRunningTime="2026-10-31 03:01:15.825065599 +0000 UTC m=+357.505591723" Oct 31 03:01:17.752894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:17.752864 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-j7fhl" Oct 31 03:01:17.817234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:17.817201 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" event={"ID":"5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7","Type":"ContainerStarted","Data":"a55f8bf2c5d2ec14977ba860b96c1247d95b4d538b6a70e520fb3e102e7e1741"} Oct 31 03:01:17.817477 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:17.817450 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" Oct 31 03:01:17.841877 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:17.841818 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" podStartSLOduration=1.699900023 podStartE2EDuration="4.841804296s" totalImagesPullingTime="3.141904273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:17.840573773 +0000 UTC m=+359.521099902" watchObservedRunningTime="2026-10-31 03:01:17.841804296 +0000 UTC m=+359.522330423" Oct 31 03:01:23.707652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.707616 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-rrspv"] Oct 31 03:01:23.715008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.714980 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rrspv" Oct 31 03:01:23.721491 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.721465 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rrspv"] Oct 31 03:01:23.726989 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.726966 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bf97k\"" Oct 31 03:01:23.824219 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.824189 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" Oct 31 03:01:23.884370 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.884337 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8f76806-1ccc-4406-82ac-488223652b0a-bound-sa-token\") pod \"cert-manager-597b95cb84-rrspv\" (UID: \"a8f76806-1ccc-4406-82ac-488223652b0a\") " pod="cert-manager/cert-manager-597b95cb84-rrspv" Oct 31 03:01:23.884529 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.884401 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwq2\" (UniqueName: \"kubernetes.io/projected/a8f76806-1ccc-4406-82ac-488223652b0a-kube-api-access-njwq2\") pod \"cert-manager-597b95cb84-rrspv\" (UID: \"a8f76806-1ccc-4406-82ac-488223652b0a\") " pod="cert-manager/cert-manager-597b95cb84-rrspv" Oct 31 03:01:23.985265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.985181 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwq2\" (UniqueName: \"kubernetes.io/projected/a8f76806-1ccc-4406-82ac-488223652b0a-kube-api-access-njwq2\") pod \"cert-manager-597b95cb84-rrspv\" (UID: \"a8f76806-1ccc-4406-82ac-488223652b0a\") " pod="cert-manager/cert-manager-597b95cb84-rrspv" Oct 31 03:01:23.985265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.985264 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8f76806-1ccc-4406-82ac-488223652b0a-bound-sa-token\") pod \"cert-manager-597b95cb84-rrspv\" (UID: \"a8f76806-1ccc-4406-82ac-488223652b0a\") " pod="cert-manager/cert-manager-597b95cb84-rrspv" Oct 31 03:01:23.998478 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.998452 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8f76806-1ccc-4406-82ac-488223652b0a-bound-sa-token\") pod \"cert-manager-597b95cb84-rrspv\" (UID: \"a8f76806-1ccc-4406-82ac-488223652b0a\") " pod="cert-manager/cert-manager-597b95cb84-rrspv" Oct 31 03:01:23.998616 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:23.998595 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwq2\" (UniqueName: \"kubernetes.io/projected/a8f76806-1ccc-4406-82ac-488223652b0a-kube-api-access-njwq2\") pod \"cert-manager-597b95cb84-rrspv\" (UID: \"a8f76806-1ccc-4406-82ac-488223652b0a\") " pod="cert-manager/cert-manager-597b95cb84-rrspv" Oct 31 03:01:24.029383 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:24.029359 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rrspv" Oct 31 03:01:24.319168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:24.319143 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rrspv"] Oct 31 03:01:24.320175 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:01:24.320145 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8f76806_1ccc_4406_82ac_488223652b0a.slice/crio-8d7e75347226d53e66eedffbf82d48626805ca63d77a62d75f67281b4833aca3 WatchSource:0}: Error finding container 8d7e75347226d53e66eedffbf82d48626805ca63d77a62d75f67281b4833aca3: Status 404 returned error can't find the container with id 8d7e75347226d53e66eedffbf82d48626805ca63d77a62d75f67281b4833aca3 Oct 31 03:01:24.854509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:24.854472 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rrspv" event={"ID":"a8f76806-1ccc-4406-82ac-488223652b0a","Type":"ContainerStarted","Data":"8818be8d962e03efaaaae56e7d8f7b961281af88869decdb612967913467ce26"} Oct 31 03:01:24.854509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:24.854521 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rrspv" event={"ID":"a8f76806-1ccc-4406-82ac-488223652b0a","Type":"ContainerStarted","Data":"8d7e75347226d53e66eedffbf82d48626805ca63d77a62d75f67281b4833aca3"} Oct 31 03:01:24.877568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:24.877519 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-rrspv" podStartSLOduration=1.8775065469999999 podStartE2EDuration="1.877506547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:24.875658092 +0000 UTC m=+366.556184222" watchObservedRunningTime="2026-10-31 03:01:24.877506547 +0000 UTC m=+366.558032674" Oct 31 03:01:27.683497 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.683458 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd"] Oct 31 03:01:27.687274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.687256 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:01:27.693994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.693922 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-swfhz\"" Oct 31 03:01:27.696016 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.695997 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 03:01:27.696016 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.696013 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 03:01:27.698302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.698281 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd"] Oct 31 03:01:27.820814 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.820778 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-68fpd\" (UID: \"895c6770-36a4-4583-86c1-2e5b7b5d8ee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:01:27.820991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.820909 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzcq\" (UniqueName: \"kubernetes.io/projected/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-kube-api-access-wbzcq\") pod \"kuadrant-operator-controller-manager-cf8469d7b-68fpd\" (UID: \"895c6770-36a4-4583-86c1-2e5b7b5d8ee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:01:27.921583 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.921540 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-68fpd\" (UID: \"895c6770-36a4-4583-86c1-2e5b7b5d8ee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:01:27.921742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.921651 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzcq\" (UniqueName: \"kubernetes.io/projected/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-kube-api-access-wbzcq\") pod \"kuadrant-operator-controller-manager-cf8469d7b-68fpd\" (UID: \"895c6770-36a4-4583-86c1-2e5b7b5d8ee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:01:27.921947 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.921927 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-68fpd\" (UID: \"895c6770-36a4-4583-86c1-2e5b7b5d8ee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:01:27.964555 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:27.964489 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzcq\" (UniqueName: \"kubernetes.io/projected/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-kube-api-access-wbzcq\") pod \"kuadrant-operator-controller-manager-cf8469d7b-68fpd\" (UID: \"895c6770-36a4-4583-86c1-2e5b7b5d8ee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:01:28.002051 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:28.002004 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:01:28.303524 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:28.303495 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd"] Oct 31 03:01:28.303919 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:01:28.303893 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895c6770_36a4_4583_86c1_2e5b7b5d8ee3.slice/crio-2b30107c314658800a5a6b811eedae69319fc5a0448b9ae6c002060e769e0eb4 WatchSource:0}: Error finding container 2b30107c314658800a5a6b811eedae69319fc5a0448b9ae6c002060e769e0eb4: Status 404 returned error can't find the container with id 2b30107c314658800a5a6b811eedae69319fc5a0448b9ae6c002060e769e0eb4 Oct 31 03:01:28.875644 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:28.875605 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" event={"ID":"895c6770-36a4-4583-86c1-2e5b7b5d8ee3","Type":"ContainerStarted","Data":"2b30107c314658800a5a6b811eedae69319fc5a0448b9ae6c002060e769e0eb4"} Oct 31 03:01:32.143625 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:01:32.143536 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:01:32.258877 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.258838 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm"] Oct 31 03:01:32.262685 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.262663 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.276631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.276357 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 03:01:32.276631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.276426 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 03:01:32.276631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.276540 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-skvb5\"" Oct 31 03:01:32.277298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.277257 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm"] Oct 31 03:01:32.361563 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.361531 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bfd4f57-1369-425c-a3f4-d762e147c354-tmp\") pod \"kube-auth-proxy-5d9fdc54f-j5svm\" (UID: \"7bfd4f57-1369-425c-a3f4-d762e147c354\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.361757 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.361568 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bfd4f57-1369-425c-a3f4-d762e147c354-tls-certs\") pod \"kube-auth-proxy-5d9fdc54f-j5svm\" (UID: \"7bfd4f57-1369-425c-a3f4-d762e147c354\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.361757 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.361715 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9n2\" (UniqueName: \"kubernetes.io/projected/7bfd4f57-1369-425c-a3f4-d762e147c354-kube-api-access-tm9n2\") pod \"kube-auth-proxy-5d9fdc54f-j5svm\" (UID: \"7bfd4f57-1369-425c-a3f4-d762e147c354\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.462968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.462930 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bfd4f57-1369-425c-a3f4-d762e147c354-tmp\") pod \"kube-auth-proxy-5d9fdc54f-j5svm\" (UID: \"7bfd4f57-1369-425c-a3f4-d762e147c354\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.463182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.463090 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bfd4f57-1369-425c-a3f4-d762e147c354-tls-certs\") pod \"kube-auth-proxy-5d9fdc54f-j5svm\" (UID: \"7bfd4f57-1369-425c-a3f4-d762e147c354\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.463252 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.463186 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9n2\" (UniqueName: \"kubernetes.io/projected/7bfd4f57-1369-425c-a3f4-d762e147c354-kube-api-access-tm9n2\") pod \"kube-auth-proxy-5d9fdc54f-j5svm\" (UID: \"7bfd4f57-1369-425c-a3f4-d762e147c354\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.465259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.465236 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bfd4f57-1369-425c-a3f4-d762e147c354-tmp\") pod \"kube-auth-proxy-5d9fdc54f-j5svm\" (UID: \"7bfd4f57-1369-425c-a3f4-d762e147c354\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.465426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.465407 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bfd4f57-1369-425c-a3f4-d762e147c354-tls-certs\") pod \"kube-auth-proxy-5d9fdc54f-j5svm\" (UID: \"7bfd4f57-1369-425c-a3f4-d762e147c354\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.480607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.480556 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9n2\" (UniqueName: \"kubernetes.io/projected/7bfd4f57-1369-425c-a3f4-d762e147c354-kube-api-access-tm9n2\") pod \"kube-auth-proxy-5d9fdc54f-j5svm\" (UID: \"7bfd4f57-1369-425c-a3f4-d762e147c354\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.585010 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.584975 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" Oct 31 03:01:32.887639 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.887613 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm"] Oct 31 03:01:32.889184 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:01:32.889150 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfd4f57_1369_425c_a3f4_d762e147c354.slice/crio-5c86738a647f5a447b6b1dedea06a8f187522447892a89b95de48211696f5557 WatchSource:0}: Error finding container 5c86738a647f5a447b6b1dedea06a8f187522447892a89b95de48211696f5557: Status 404 returned error can't find the container with id 5c86738a647f5a447b6b1dedea06a8f187522447892a89b95de48211696f5557 Oct 31 03:01:32.899054 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:32.899018 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" event={"ID":"7bfd4f57-1369-425c-a3f4-d762e147c354","Type":"ContainerStarted","Data":"5c86738a647f5a447b6b1dedea06a8f187522447892a89b95de48211696f5557"} Oct 31 03:01:34.343064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:34.343017 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf"] Oct 31 03:01:34.346628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:34.346602 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf" Oct 31 03:01:34.358072 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:34.358025 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6gxz4\"" Oct 31 03:01:34.369073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:34.369021 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 03:01:34.369482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:34.369462 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf"] Oct 31 03:01:34.383896 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:34.383857 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk5l4\" (UniqueName: \"kubernetes.io/projected/feb0e371-5edb-41d6-a003-9e4adf0853b9-kube-api-access-fk5l4\") pod \"dns-operator-controller-manager-8557cbdb84-lblcf\" (UID: \"feb0e371-5edb-41d6-a003-9e4adf0853b9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf" Oct 31 03:01:34.486229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:34.486191 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk5l4\" (UniqueName: \"kubernetes.io/projected/feb0e371-5edb-41d6-a003-9e4adf0853b9-kube-api-access-fk5l4\") pod \"dns-operator-controller-manager-8557cbdb84-lblcf\" (UID: \"feb0e371-5edb-41d6-a003-9e4adf0853b9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf" Oct 31 03:01:34.511876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:34.511840 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk5l4\" (UniqueName: \"kubernetes.io/projected/feb0e371-5edb-41d6-a003-9e4adf0853b9-kube-api-access-fk5l4\") pod \"dns-operator-controller-manager-8557cbdb84-lblcf\" (UID: \"feb0e371-5edb-41d6-a003-9e4adf0853b9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf" Oct 31 03:01:34.670777 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:34.670741 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf" Oct 31 03:01:34.963460 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:34.963049 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf"] Oct 31 03:01:34.965495 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:01:34.965463 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb0e371_5edb_41d6_a003_9e4adf0853b9.slice/crio-24d58d355d3a58db2f279b4204134dad5cd687d92c1d71ae5bb5f30e3b2a14d9 WatchSource:0}: Error finding container 24d58d355d3a58db2f279b4204134dad5cd687d92c1d71ae5bb5f30e3b2a14d9: Status 404 returned error can't find the container with id 24d58d355d3a58db2f279b4204134dad5cd687d92c1d71ae5bb5f30e3b2a14d9 Oct 31 03:01:35.922861 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:35.922807 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf" event={"ID":"feb0e371-5edb-41d6-a003-9e4adf0853b9","Type":"ContainerStarted","Data":"24d58d355d3a58db2f279b4204134dad5cd687d92c1d71ae5bb5f30e3b2a14d9"} Oct 31 03:01:36.932248 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:36.932196 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" event={"ID":"895c6770-36a4-4583-86c1-2e5b7b5d8ee3","Type":"ContainerStarted","Data":"1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70"} Oct 31 03:01:36.932636 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:36.932279 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:01:36.933826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:36.933801 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" event={"ID":"7bfd4f57-1369-425c-a3f4-d762e147c354","Type":"ContainerStarted","Data":"a2b1eb944ea0153ae2edb522da154b70b0fffbc6708d95e60820cf940ab7acba"} Oct 31 03:01:36.965747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:36.965684 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" podStartSLOduration=2.420524169 podStartE2EDuration="9.965664515s" totalImagesPullingTime="7.545140346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:36.960413741 +0000 UTC m=+378.640939868" watchObservedRunningTime="2026-10-31 03:01:36.965664515 +0000 UTC m=+378.646190645" Oct 31 03:01:36.983139 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:36.983087 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-j5svm" podStartSLOduration=1.953517376 podStartE2EDuration="4.983071529s" totalImagesPullingTime="3.029554153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:36.980685683 +0000 UTC m=+378.661211813" watchObservedRunningTime="2026-10-31 03:01:36.983071529 +0000 UTC m=+378.663597653" Oct 31 03:01:37.940952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:37.940917 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf" event={"ID":"feb0e371-5edb-41d6-a003-9e4adf0853b9","Type":"ContainerStarted","Data":"9c784c81f39b2f3f5f9907f24a585fe3ee9607dc2951677e9952f89765d5dd95"} Oct 31 03:01:37.965222 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:37.965177 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf" podStartSLOduration=1.5271613689999999 podStartE2EDuration="3.965163945s" totalImagesPullingTime="2.438002576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:37.962931856 +0000 UTC m=+379.643457984" watchObservedRunningTime="2026-10-31 03:01:37.965163945 +0000 UTC m=+379.645690071" Oct 31 03:01:38.945802 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:38.945770 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf" Oct 31 03:01:47.943317 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:47.943284 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:01:48.308388 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.308309 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc"] Oct 31 03:01:48.311252 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.311236 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:01:48.326645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.326621 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc"] Oct 31 03:01:48.413681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.413641 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4hf\" (UniqueName: \"kubernetes.io/projected/92f6df64-c757-4734-9cb0-59d3558b538b-kube-api-access-dp4hf\") pod \"kuadrant-operator-controller-manager-678d87766f-xgwpc\" (UID: \"92f6df64-c757-4734-9cb0-59d3558b538b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:01:48.413819 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.413713 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92f6df64-c757-4734-9cb0-59d3558b538b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xgwpc\" (UID: \"92f6df64-c757-4734-9cb0-59d3558b538b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:01:48.515105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.515058 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92f6df64-c757-4734-9cb0-59d3558b538b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xgwpc\" (UID: \"92f6df64-c757-4734-9cb0-59d3558b538b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:01:48.515382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.515205 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4hf\" (UniqueName: \"kubernetes.io/projected/92f6df64-c757-4734-9cb0-59d3558b538b-kube-api-access-dp4hf\") pod \"kuadrant-operator-controller-manager-678d87766f-xgwpc\" (UID: \"92f6df64-c757-4734-9cb0-59d3558b538b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:01:48.515469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.515449 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92f6df64-c757-4734-9cb0-59d3558b538b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xgwpc\" (UID: \"92f6df64-c757-4734-9cb0-59d3558b538b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:01:48.529793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.529765 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4hf\" (UniqueName: \"kubernetes.io/projected/92f6df64-c757-4734-9cb0-59d3558b538b-kube-api-access-dp4hf\") pod \"kuadrant-operator-controller-manager-678d87766f-xgwpc\" (UID: \"92f6df64-c757-4734-9cb0-59d3558b538b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:01:48.622904 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.622817 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:01:48.910709 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:48.910685 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc"] Oct 31 03:01:48.912078 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:01:48.912028 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f6df64_c757_4734_9cb0_59d3558b538b.slice/crio-95c252cb35f3603347f261eb9a90b52a7cce597d8c7deaa504fe74b20c58ea4c WatchSource:0}: Error finding container 95c252cb35f3603347f261eb9a90b52a7cce597d8c7deaa504fe74b20c58ea4c: Status 404 returned error can't find the container with id 95c252cb35f3603347f261eb9a90b52a7cce597d8c7deaa504fe74b20c58ea4c Oct 31 03:01:49.001573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:49.001541 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" event={"ID":"92f6df64-c757-4734-9cb0-59d3558b538b","Type":"ContainerStarted","Data":"a095de909e2b640036ec8b4b4ff44f580dcbd1bda514598426d2f9e0177f6a66"} Oct 31 03:01:49.001918 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:49.001593 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" event={"ID":"92f6df64-c757-4734-9cb0-59d3558b538b","Type":"ContainerStarted","Data":"95c252cb35f3603347f261eb9a90b52a7cce597d8c7deaa504fe74b20c58ea4c"} Oct 31 03:01:49.001918 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:49.001653 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:01:49.953875 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:49.953846 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lblcf" Oct 31 03:01:49.978950 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:01:49.978899 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" podStartSLOduration=1.978884458 podStartE2EDuration="1.978884458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:49.043756614 +0000 UTC m=+390.724282742" watchObservedRunningTime="2026-10-31 03:01:49.978884458 +0000 UTC m=+391.659410594" Oct 31 03:02:00.009333 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:00.009299 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:02:00.075380 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:00.075348 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd"] Oct 31 03:02:00.075596 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:00.075546 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" podUID="895c6770-36a4-4583-86c1-2e5b7b5d8ee3" containerName="manager" containerID="cri-o://1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70" gracePeriod=10 Oct 31 03:02:00.318783 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:00.318754 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:02:00.423665 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:00.423630 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-extensions-socket-volume\") pod \"895c6770-36a4-4583-86c1-2e5b7b5d8ee3\" (UID: \"895c6770-36a4-4583-86c1-2e5b7b5d8ee3\") " Oct 31 03:02:00.423828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:00.423672 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-kube-api-access-wbzcq\" (UniqueName: \"kubernetes.io/projected/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-kube-api-access-wbzcq\") pod \"895c6770-36a4-4583-86c1-2e5b7b5d8ee3\" (UID: \"895c6770-36a4-4583-86c1-2e5b7b5d8ee3\") " Oct 31 03:02:00.424184 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:00.424162 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-extensions-socket-volume" pod "895c6770-36a4-4583-86c1-2e5b7b5d8ee3" (UID: "895c6770-36a4-4583-86c1-2e5b7b5d8ee3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:02:00.425920 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:00.425892 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-kube-api-access-wbzcq" pod "895c6770-36a4-4583-86c1-2e5b7b5d8ee3" (UID: "895c6770-36a4-4583-86c1-2e5b7b5d8ee3"). InnerVolumeSpecName "kube-api-access-wbzcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:02:00.525327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:00.525295 4456 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-extensions-socket-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:00.525327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:00.525323 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbzcq\" (UniqueName: \"kubernetes.io/projected/895c6770-36a4-4583-86c1-2e5b7b5d8ee3-kube-api-access-wbzcq\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:01.084575 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.084478 4456 generic.go:356] "Generic (PLEG): container finished" podID="895c6770-36a4-4583-86c1-2e5b7b5d8ee3" containerID="1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70" exitCode=0 Oct 31 03:02:01.084575 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.084536 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" Oct 31 03:02:01.085092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.084566 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" event={"ID":"895c6770-36a4-4583-86c1-2e5b7b5d8ee3","Type":"ContainerDied","Data":"1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70"} Oct 31 03:02:01.085092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.084624 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd" event={"ID":"895c6770-36a4-4583-86c1-2e5b7b5d8ee3","Type":"ContainerDied","Data":"2b30107c314658800a5a6b811eedae69319fc5a0448b9ae6c002060e769e0eb4"} Oct 31 03:02:01.085092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.084648 4456 scope.go:122] "RemoveContainer" containerID="1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70" Oct 31 03:02:01.096444 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.096387 4456 scope.go:122] "RemoveContainer" containerID="1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70" Oct 31 03:02:01.096812 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:02:01.096791 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70\": container with ID starting with 1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70 not found: ID does not exist" containerID="1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70" Oct 31 03:02:01.096920 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.096819 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70"} err="failed to get container status \"1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70\": rpc error: code = NotFound desc = could not find container \"1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70\": container with ID starting with 1fbc44e78a874dc2343839b181a4ccf1aacaef19b884e7a0e69ce1641b963d70 not found: ID does not exist" Oct 31 03:02:01.119402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.119376 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd"] Oct 31 03:02:01.127236 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.127213 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-68fpd"] Oct 31 03:02:01.356129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.355993 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-b5h2b"] Oct 31 03:02:01.356517 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.356498 4456 cpu_manager.go:414] "containerMap: removing container" podUID="895c6770-36a4-4583-86c1-2e5b7b5d8ee3" containerName="manager" Oct 31 03:02:01.356605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.356520 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895c6770-36a4-4583-86c1-2e5b7b5d8ee3" containerName="manager" Oct 31 03:02:01.356659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.356632 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="895c6770-36a4-4583-86c1-2e5b7b5d8ee3" containerName="manager" Oct 31 03:02:01.361082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.361064 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:01.367172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.367115 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tdjqk\"" Oct 31 03:02:01.368815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.368794 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 03:02:01.369434 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.369413 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-b5h2b"] Oct 31 03:02:01.434015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.433984 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m5t9\" (UniqueName: \"kubernetes.io/projected/5f0d1332-ddc7-46f6-8116-962ec0e35531-kube-api-access-4m5t9\") pod \"limitador-limitador-74dfcfb894-b5h2b\" (UID: \"5f0d1332-ddc7-46f6-8116-962ec0e35531\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:01.434015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.434022 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5f0d1332-ddc7-46f6-8116-962ec0e35531-config-file\") pod \"limitador-limitador-74dfcfb894-b5h2b\" (UID: \"5f0d1332-ddc7-46f6-8116-962ec0e35531\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:01.451652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.451626 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-b5h2b"] Oct 31 03:02:01.535402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.535367 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m5t9\" (UniqueName: \"kubernetes.io/projected/5f0d1332-ddc7-46f6-8116-962ec0e35531-kube-api-access-4m5t9\") pod \"limitador-limitador-74dfcfb894-b5h2b\" (UID: \"5f0d1332-ddc7-46f6-8116-962ec0e35531\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:01.535569 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.535405 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5f0d1332-ddc7-46f6-8116-962ec0e35531-config-file\") pod \"limitador-limitador-74dfcfb894-b5h2b\" (UID: \"5f0d1332-ddc7-46f6-8116-962ec0e35531\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:01.536081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.536059 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5f0d1332-ddc7-46f6-8116-962ec0e35531-config-file\") pod \"limitador-limitador-74dfcfb894-b5h2b\" (UID: \"5f0d1332-ddc7-46f6-8116-962ec0e35531\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:01.553782 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.553755 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m5t9\" (UniqueName: \"kubernetes.io/projected/5f0d1332-ddc7-46f6-8116-962ec0e35531-kube-api-access-4m5t9\") pod \"limitador-limitador-74dfcfb894-b5h2b\" (UID: \"5f0d1332-ddc7-46f6-8116-962ec0e35531\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:01.674454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.674430 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:01.951371 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:01.951342 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-b5h2b"] Oct 31 03:02:01.951588 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:02:01.951564 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0d1332_ddc7_46f6_8116_962ec0e35531.slice/crio-dade9bde15424d8073ec65f9e6d87e4968dc634b27eb67aa583c53d6e9aad1d9 WatchSource:0}: Error finding container dade9bde15424d8073ec65f9e6d87e4968dc634b27eb67aa583c53d6e9aad1d9: Status 404 returned error can't find the container with id dade9bde15424d8073ec65f9e6d87e4968dc634b27eb67aa583c53d6e9aad1d9 Oct 31 03:02:02.092286 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:02.092248 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" event={"ID":"5f0d1332-ddc7-46f6-8116-962ec0e35531","Type":"ContainerStarted","Data":"dade9bde15424d8073ec65f9e6d87e4968dc634b27eb67aa583c53d6e9aad1d9"} Oct 31 03:02:02.132955 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:02:02.132917 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:02:02.435500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:02.435462 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895c6770-36a4-4583-86c1-2e5b7b5d8ee3" path="/var/lib/kubelet/pods/895c6770-36a4-4583-86c1-2e5b7b5d8ee3/volumes" Oct 31 03:02:04.107654 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:04.107612 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" event={"ID":"5f0d1332-ddc7-46f6-8116-962ec0e35531","Type":"ContainerStarted","Data":"2fd948a01d0104a6dcac29285485839b8da24312eecc0ef93566206bde877bb9"} Oct 31 03:02:04.108157 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:04.107699 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:04.129411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:04.129361 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" podStartSLOduration=1.667408869 podStartE2EDuration="3.129346397s" totalImagesPullingTime="1.461937528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:02:04.12819103 +0000 UTC m=+405.808717158" watchObservedRunningTime="2026-10-31 03:02:04.129346397 +0000 UTC m=+405.809872523" Oct 31 03:02:15.114403 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:15.114370 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:30.149470 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.149432 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qprkr"] Oct 31 03:02:30.156882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.156859 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" Oct 31 03:02:30.166125 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.166100 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wtbdb\"" Oct 31 03:02:30.167192 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.167168 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qprkr"] Oct 31 03:02:30.169298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.169276 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 03:02:30.169422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.169321 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 03:02:30.170123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.170102 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs"] Oct 31 03:02:30.173369 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.173350 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:02:30.179402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.179235 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/462a67f6-9d05-41d3-81a4-6729996976a1-cert\") pod \"kserve-controller-manager-776cc66d47-qprkr\" (UID: \"462a67f6-9d05-41d3-81a4-6729996976a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" Oct 31 03:02:30.179402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.179272 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f86hh\" (UniqueName: \"kubernetes.io/projected/462a67f6-9d05-41d3-81a4-6729996976a1-kube-api-access-f86hh\") pod \"kserve-controller-manager-776cc66d47-qprkr\" (UID: \"462a67f6-9d05-41d3-81a4-6729996976a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" Oct 31 03:02:30.188090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.188028 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs"] Oct 31 03:02:30.225500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.225453 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2q6tj\"" Oct 31 03:02:30.225500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.225484 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 03:02:30.225500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.225477 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 03:02:30.280650 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.280620 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1b2a81a-0acd-43cb-b216-49b14b24b450-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-s5xvs\" (UID: \"c1b2a81a-0acd-43cb-b216-49b14b24b450\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:02:30.280811 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.280735 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/462a67f6-9d05-41d3-81a4-6729996976a1-cert\") pod \"kserve-controller-manager-776cc66d47-qprkr\" (UID: \"462a67f6-9d05-41d3-81a4-6729996976a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" Oct 31 03:02:30.280811 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.280777 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f86hh\" (UniqueName: \"kubernetes.io/projected/462a67f6-9d05-41d3-81a4-6729996976a1-kube-api-access-f86hh\") pod \"kserve-controller-manager-776cc66d47-qprkr\" (UID: \"462a67f6-9d05-41d3-81a4-6729996976a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" Oct 31 03:02:30.280905 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.280809 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s4l5\" (UniqueName: \"kubernetes.io/projected/c1b2a81a-0acd-43cb-b216-49b14b24b450-kube-api-access-4s4l5\") pod \"llmisvc-controller-manager-869d5dcc5d-s5xvs\" (UID: \"c1b2a81a-0acd-43cb-b216-49b14b24b450\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:02:30.283348 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.283322 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/462a67f6-9d05-41d3-81a4-6729996976a1-cert\") pod \"kserve-controller-manager-776cc66d47-qprkr\" (UID: \"462a67f6-9d05-41d3-81a4-6729996976a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" Oct 31 03:02:30.299354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.299327 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f86hh\" (UniqueName: \"kubernetes.io/projected/462a67f6-9d05-41d3-81a4-6729996976a1-kube-api-access-f86hh\") pod \"kserve-controller-manager-776cc66d47-qprkr\" (UID: \"462a67f6-9d05-41d3-81a4-6729996976a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" Oct 31 03:02:30.381440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.381406 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s4l5\" (UniqueName: \"kubernetes.io/projected/c1b2a81a-0acd-43cb-b216-49b14b24b450-kube-api-access-4s4l5\") pod \"llmisvc-controller-manager-869d5dcc5d-s5xvs\" (UID: \"c1b2a81a-0acd-43cb-b216-49b14b24b450\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:02:30.381602 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.381452 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1b2a81a-0acd-43cb-b216-49b14b24b450-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-s5xvs\" (UID: \"c1b2a81a-0acd-43cb-b216-49b14b24b450\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:02:30.383774 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.383742 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1b2a81a-0acd-43cb-b216-49b14b24b450-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-s5xvs\" (UID: \"c1b2a81a-0acd-43cb-b216-49b14b24b450\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:02:30.395260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.395234 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s4l5\" (UniqueName: \"kubernetes.io/projected/c1b2a81a-0acd-43cb-b216-49b14b24b450-kube-api-access-4s4l5\") pod \"llmisvc-controller-manager-869d5dcc5d-s5xvs\" (UID: \"c1b2a81a-0acd-43cb-b216-49b14b24b450\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:02:30.473472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.473437 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" Oct 31 03:02:30.485568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.485543 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:02:30.751456 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.751424 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qprkr"] Oct 31 03:02:30.752028 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:02:30.751999 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462a67f6_9d05_41d3_81a4_6729996976a1.slice/crio-0ec6e2d286daf5c43cc9cde9015542bfbe41856cb7ba0a7871344570e7badd0e WatchSource:0}: Error finding container 0ec6e2d286daf5c43cc9cde9015542bfbe41856cb7ba0a7871344570e7badd0e: Status 404 returned error can't find the container with id 0ec6e2d286daf5c43cc9cde9015542bfbe41856cb7ba0a7871344570e7badd0e Oct 31 03:02:30.767184 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.767161 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs"] Oct 31 03:02:30.767944 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:02:30.767920 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc1b2a81a_0acd_43cb_b216_49b14b24b450.slice/crio-a9d26cb4a1c6d63e3a3073457cd4d6c3248460ab632ca1f31df7a83654b01ff3 WatchSource:0}: Error finding container a9d26cb4a1c6d63e3a3073457cd4d6c3248460ab632ca1f31df7a83654b01ff3: Status 404 returned error can't find the container with id a9d26cb4a1c6d63e3a3073457cd4d6c3248460ab632ca1f31df7a83654b01ff3 Oct 31 03:02:30.819088 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.819056 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-b5h2b"] Oct 31 03:02:30.819319 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:30.819285 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" podUID="5f0d1332-ddc7-46f6-8116-962ec0e35531" containerName="limitador" containerID="cri-o://2fd948a01d0104a6dcac29285485839b8da24312eecc0ef93566206bde877bb9" gracePeriod=30 Oct 31 03:02:31.256939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.256907 4456 generic.go:356] "Generic (PLEG): container finished" podID="5f0d1332-ddc7-46f6-8116-962ec0e35531" containerID="2fd948a01d0104a6dcac29285485839b8da24312eecc0ef93566206bde877bb9" exitCode=0 Oct 31 03:02:31.257438 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.257015 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" event={"ID":"5f0d1332-ddc7-46f6-8116-962ec0e35531","Type":"ContainerDied","Data":"2fd948a01d0104a6dcac29285485839b8da24312eecc0ef93566206bde877bb9"} Oct 31 03:02:31.258525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.258502 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" event={"ID":"c1b2a81a-0acd-43cb-b216-49b14b24b450","Type":"ContainerStarted","Data":"a9d26cb4a1c6d63e3a3073457cd4d6c3248460ab632ca1f31df7a83654b01ff3"} Oct 31 03:02:31.259910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.259885 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" event={"ID":"462a67f6-9d05-41d3-81a4-6729996976a1","Type":"ContainerStarted","Data":"0ec6e2d286daf5c43cc9cde9015542bfbe41856cb7ba0a7871344570e7badd0e"} Oct 31 03:02:31.391424 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.391352 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:31.494647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.494610 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0d1332-ddc7-46f6-8116-962ec0e35531-kube-api-access-4m5t9\" (UniqueName: \"kubernetes.io/projected/5f0d1332-ddc7-46f6-8116-962ec0e35531-kube-api-access-4m5t9\") pod \"5f0d1332-ddc7-46f6-8116-962ec0e35531\" (UID: \"5f0d1332-ddc7-46f6-8116-962ec0e35531\") " Oct 31 03:02:31.494832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.494665 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0d1332-ddc7-46f6-8116-962ec0e35531-config-file\" (UniqueName: \"kubernetes.io/configmap/5f0d1332-ddc7-46f6-8116-962ec0e35531-config-file\") pod \"5f0d1332-ddc7-46f6-8116-962ec0e35531\" (UID: \"5f0d1332-ddc7-46f6-8116-962ec0e35531\") " Oct 31 03:02:31.495293 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.495259 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0d1332-ddc7-46f6-8116-962ec0e35531-config-file" pod "5f0d1332-ddc7-46f6-8116-962ec0e35531" (UID: "5f0d1332-ddc7-46f6-8116-962ec0e35531"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:02:31.497203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.497169 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0d1332-ddc7-46f6-8116-962ec0e35531-kube-api-access-4m5t9" pod "5f0d1332-ddc7-46f6-8116-962ec0e35531" (UID: "5f0d1332-ddc7-46f6-8116-962ec0e35531"). InnerVolumeSpecName "kube-api-access-4m5t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:02:31.596024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.595984 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m5t9\" (UniqueName: \"kubernetes.io/projected/5f0d1332-ddc7-46f6-8116-962ec0e35531-kube-api-access-4m5t9\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:31.596024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:31.596017 4456 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5f0d1332-ddc7-46f6-8116-962ec0e35531-config-file\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:32.141957 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:02:32.141914 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:02:32.272360 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:32.272124 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" Oct 31 03:02:32.272360 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:32.272158 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-b5h2b" event={"ID":"5f0d1332-ddc7-46f6-8116-962ec0e35531","Type":"ContainerDied","Data":"dade9bde15424d8073ec65f9e6d87e4968dc634b27eb67aa583c53d6e9aad1d9"} Oct 31 03:02:32.272360 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:32.272226 4456 scope.go:122] "RemoveContainer" containerID="2fd948a01d0104a6dcac29285485839b8da24312eecc0ef93566206bde877bb9" Oct 31 03:02:32.305972 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:32.305937 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-b5h2b"] Oct 31 03:02:32.308997 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:32.308969 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-b5h2b"] Oct 31 03:02:32.436897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:32.436863 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0d1332-ddc7-46f6-8116-962ec0e35531" path="/var/lib/kubelet/pods/5f0d1332-ddc7-46f6-8116-962ec0e35531/volumes" Oct 31 03:02:33.280235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:33.280201 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" event={"ID":"c1b2a81a-0acd-43cb-b216-49b14b24b450","Type":"ContainerStarted","Data":"d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1"} Oct 31 03:02:33.280608 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:33.280278 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:02:33.281912 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:33.281887 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" event={"ID":"462a67f6-9d05-41d3-81a4-6729996976a1","Type":"ContainerStarted","Data":"7fd32d2855eefceaa66490402f2be0e585dce306476f36a7cf05dc6c42fa7236"} Oct 31 03:02:33.282026 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:33.281990 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" Oct 31 03:02:33.305321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:33.305271 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" podStartSLOduration=0.873638031 podStartE2EDuration="3.305256735s" totalImagesPullingTime="2.431618704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:02:33.301460385 +0000 UTC m=+434.981986572" watchObservedRunningTime="2026-10-31 03:02:33.305256735 +0000 UTC m=+434.985782862" Oct 31 03:02:33.325352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:33.325300 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" podStartSLOduration=1.1458614360000001 podStartE2EDuration="3.325287363s" totalImagesPullingTime="2.179425927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:02:33.323055349 +0000 UTC m=+435.003581473" watchObservedRunningTime="2026-10-31 03:02:33.325287363 +0000 UTC m=+435.005813489" Oct 31 03:02:36.813447 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.813407 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9"] Oct 31 03:02:36.813879 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.813864 4456 cpu_manager.go:414] "containerMap: removing container" podUID="5f0d1332-ddc7-46f6-8116-962ec0e35531" containerName="limitador" Oct 31 03:02:36.813926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.813882 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0d1332-ddc7-46f6-8116-962ec0e35531" containerName="limitador" Oct 31 03:02:36.813995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.813986 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0d1332-ddc7-46f6-8116-962ec0e35531" containerName="limitador" Oct 31 03:02:36.817181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.817164 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" Oct 31 03:02:36.825478 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.822698 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 03:02:36.825478 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.824899 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:02:36.825478 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.824964 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rprw2\"" Oct 31 03:02:36.825478 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.825022 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:02:36.825478 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.825428 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9"] Oct 31 03:02:36.847395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.847359 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqmt\" (UniqueName: \"kubernetes.io/projected/7f66efa3-2b56-47c3-9c5a-05b9e597c338-kube-api-access-mpqmt\") pod \"postgres-7cdbf6dc44-9t8b9\" (UID: \"7f66efa3-2b56-47c3-9c5a-05b9e597c338\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" Oct 31 03:02:36.847697 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.847680 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7f66efa3-2b56-47c3-9c5a-05b9e597c338-data\") pod \"postgres-7cdbf6dc44-9t8b9\" (UID: \"7f66efa3-2b56-47c3-9c5a-05b9e597c338\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" Oct 31 03:02:36.949001 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.948958 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7f66efa3-2b56-47c3-9c5a-05b9e597c338-data\") pod \"postgres-7cdbf6dc44-9t8b9\" (UID: \"7f66efa3-2b56-47c3-9c5a-05b9e597c338\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" Oct 31 03:02:36.949230 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.949057 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqmt\" (UniqueName: \"kubernetes.io/projected/7f66efa3-2b56-47c3-9c5a-05b9e597c338-kube-api-access-mpqmt\") pod \"postgres-7cdbf6dc44-9t8b9\" (UID: \"7f66efa3-2b56-47c3-9c5a-05b9e597c338\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" Oct 31 03:02:36.949530 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.949502 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7f66efa3-2b56-47c3-9c5a-05b9e597c338-data\") pod \"postgres-7cdbf6dc44-9t8b9\" (UID: \"7f66efa3-2b56-47c3-9c5a-05b9e597c338\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" Oct 31 03:02:36.974745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:36.974719 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqmt\" (UniqueName: \"kubernetes.io/projected/7f66efa3-2b56-47c3-9c5a-05b9e597c338-kube-api-access-mpqmt\") pod \"postgres-7cdbf6dc44-9t8b9\" (UID: \"7f66efa3-2b56-47c3-9c5a-05b9e597c338\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" Oct 31 03:02:37.131220 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:37.131146 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" Oct 31 03:02:37.409253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:37.409222 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9"] Oct 31 03:02:37.410212 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:02:37.410188 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f66efa3_2b56_47c3_9c5a_05b9e597c338.slice/crio-0b934d7912177117bcca39260f6717fc18ce613b6f8b0853f89fa010df3615fb WatchSource:0}: Error finding container 0b934d7912177117bcca39260f6717fc18ce613b6f8b0853f89fa010df3615fb: Status 404 returned error can't find the container with id 0b934d7912177117bcca39260f6717fc18ce613b6f8b0853f89fa010df3615fb Oct 31 03:02:38.310149 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:38.310110 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" event={"ID":"7f66efa3-2b56-47c3-9c5a-05b9e597c338","Type":"ContainerStarted","Data":"0b934d7912177117bcca39260f6717fc18ce613b6f8b0853f89fa010df3615fb"} Oct 31 03:02:43.343782 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:43.343743 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" event={"ID":"7f66efa3-2b56-47c3-9c5a-05b9e597c338","Type":"ContainerStarted","Data":"26ffc63cc9d30b6d3b5ad43e6b13f3ef8edd7f3b88cdc7405812a6f1bba004f5"} Oct 31 03:02:43.344214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:43.343818 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" Oct 31 03:02:43.366424 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:43.366378 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" podStartSLOduration=1.910428528 podStartE2EDuration="7.366364848s" totalImagesPullingTime="5.45593632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:02:43.364026216 +0000 UTC m=+445.044552344" watchObservedRunningTime="2026-10-31 03:02:43.366364848 +0000 UTC m=+445.046890972" Oct 31 03:02:49.379442 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:49.379414 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8b9" Oct 31 03:02:56.675757 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:56.675722 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bnt72"] Oct 31 03:02:56.898967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:56.898929 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bnt72"] Oct 31 03:02:56.899141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:56.899061 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:56.906362 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:56.906340 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 03:02:56.906482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:56.906339 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pkwql\"" Oct 31 03:02:56.906482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:56.906346 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 03:02:56.906482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:56.906413 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 03:02:57.025793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.025705 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-metrics-certs\") pod \"maas-controller-579cf7c6d-bnt72\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:57.025793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.025754 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-webhook-cert\") pod \"maas-controller-579cf7c6d-bnt72\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:57.025984 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.025903 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnntt\" (UniqueName: \"kubernetes.io/projected/4ea4d920-134e-44f9-9308-cf163c01b7b3-kube-api-access-qnntt\") pod \"maas-controller-579cf7c6d-bnt72\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:57.070293 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.070259 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64564ccd46-mqzkl"] Oct 31 03:02:57.088298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.088272 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.096751 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.096724 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64564ccd46-mqzkl"] Oct 31 03:02:57.126972 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.126939 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-metrics-certs\") pod \"maas-controller-579cf7c6d-bnt72\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:57.127151 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.126982 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-webhook-cert\") pod \"maas-controller-579cf7c6d-bnt72\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:57.127151 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.127066 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnntt\" (UniqueName: \"kubernetes.io/projected/4ea4d920-134e-44f9-9308-cf163c01b7b3-kube-api-access-qnntt\") pod \"maas-controller-579cf7c6d-bnt72\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:57.129435 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.129408 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-metrics-certs\") pod \"maas-controller-579cf7c6d-bnt72\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:57.129435 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.129432 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-webhook-cert\") pod \"maas-controller-579cf7c6d-bnt72\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:57.142327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.142304 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnntt\" (UniqueName: \"kubernetes.io/projected/4ea4d920-134e-44f9-9308-cf163c01b7b3-kube-api-access-qnntt\") pod \"maas-controller-579cf7c6d-bnt72\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:57.228097 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.228061 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-webhook-cert\") pod \"maas-controller-64564ccd46-mqzkl\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.228245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.228129 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-metrics-certs\") pod \"maas-controller-64564ccd46-mqzkl\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.228245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.228146 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh727\" (UniqueName: \"kubernetes.io/projected/5ecd0946-7571-46bf-9eb7-913be7450dd8-kube-api-access-gh727\") pod \"maas-controller-64564ccd46-mqzkl\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.231982 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.231950 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:02:57.329613 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.329581 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-metrics-certs\") pod \"maas-controller-64564ccd46-mqzkl\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.329613 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.329617 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh727\" (UniqueName: \"kubernetes.io/projected/5ecd0946-7571-46bf-9eb7-913be7450dd8-kube-api-access-gh727\") pod \"maas-controller-64564ccd46-mqzkl\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.329802 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.329674 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-webhook-cert\") pod \"maas-controller-64564ccd46-mqzkl\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.332005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.331967 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-webhook-cert\") pod \"maas-controller-64564ccd46-mqzkl\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.332005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.332005 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-metrics-certs\") pod \"maas-controller-64564ccd46-mqzkl\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.344129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.344107 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh727\" (UniqueName: \"kubernetes.io/projected/5ecd0946-7571-46bf-9eb7-913be7450dd8-kube-api-access-gh727\") pod \"maas-controller-64564ccd46-mqzkl\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.413082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.413021 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:02:57.517229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.517200 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bnt72"] Oct 31 03:02:57.518630 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:02:57.518602 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea4d920_134e_44f9_9308_cf163c01b7b3.slice/crio-6dc579a200c08b4a90196e6bb1e8096c23fbb477a579e919fb4e2f8dd1cb7c95 WatchSource:0}: Error finding container 6dc579a200c08b4a90196e6bb1e8096c23fbb477a579e919fb4e2f8dd1cb7c95: Status 404 returned error can't find the container with id 6dc579a200c08b4a90196e6bb1e8096c23fbb477a579e919fb4e2f8dd1cb7c95 Oct 31 03:02:57.710758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:57.710731 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64564ccd46-mqzkl"] Oct 31 03:02:57.711165 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:02:57.710997 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecd0946_7571_46bf_9eb7_913be7450dd8.slice/crio-7308ded5578e739a7819014b2c04a0ccd54a8720ac62e777c9134706c8723815 WatchSource:0}: Error finding container 7308ded5578e739a7819014b2c04a0ccd54a8720ac62e777c9134706c8723815: Status 404 returned error can't find the container with id 7308ded5578e739a7819014b2c04a0ccd54a8720ac62e777c9134706c8723815 Oct 31 03:02:58.427057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:58.427005 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64564ccd46-mqzkl" event={"ID":"5ecd0946-7571-46bf-9eb7-913be7450dd8","Type":"ContainerStarted","Data":"7308ded5578e739a7819014b2c04a0ccd54a8720ac62e777c9134706c8723815"} Oct 31 03:02:58.436637 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:02:58.436601 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-bnt72" event={"ID":"4ea4d920-134e-44f9-9308-cf163c01b7b3","Type":"ContainerStarted","Data":"6dc579a200c08b4a90196e6bb1e8096c23fbb477a579e919fb4e2f8dd1cb7c95"} Oct 31 03:03:01.456174 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:01.456129 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-bnt72" event={"ID":"4ea4d920-134e-44f9-9308-cf163c01b7b3","Type":"ContainerStarted","Data":"70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8"} Oct 31 03:03:01.456669 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:01.456219 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:03:01.457942 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:01.457911 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64564ccd46-mqzkl" event={"ID":"5ecd0946-7571-46bf-9eb7-913be7450dd8","Type":"ContainerStarted","Data":"fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5"} Oct 31 03:03:01.458076 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:01.458022 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:03:01.481539 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:01.481483 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-bnt72" podStartSLOduration=2.464116767 podStartE2EDuration="5.481466488s" totalImagesPullingTime="3.017349721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:03:01.478999912 +0000 UTC m=+463.159526052" watchObservedRunningTime="2026-10-31 03:03:01.481466488 +0000 UTC m=+463.161992635" Oct 31 03:03:01.507939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:01.507886 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64564ccd46-mqzkl" podStartSLOduration=1.473976068 podStartE2EDuration="4.50786283s" totalImagesPullingTime="3.033886762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:03:01.506871273 +0000 UTC m=+463.187397400" watchObservedRunningTime="2026-10-31 03:03:01.50786283 +0000 UTC m=+463.188388957" Oct 31 03:03:02.133372 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:03:02.133334 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:03:02.189509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.189479 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4"] Oct 31 03:03:02.193354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.193332 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.202265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.202241 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 03:03:02.202800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.202664 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 03:03:02.203134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.202680 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 03:03:02.203288 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.203093 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-262nc\"" Oct 31 03:03:02.206010 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.205987 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4"] Oct 31 03:03:02.381190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.381152 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8e8f2d86-6510-4386-befc-ec0c5c031127-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.381190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.381191 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8e8f2d86-6510-4386-befc-ec0c5c031127-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.381416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.381213 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8f2d86-6510-4386-befc-ec0c5c031127-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.381416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.381325 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbcm\" (UniqueName: \"kubernetes.io/projected/8e8f2d86-6510-4386-befc-ec0c5c031127-kube-api-access-pjbcm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.482363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.482328 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8e8f2d86-6510-4386-befc-ec0c5c031127-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.482363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.482363 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8e8f2d86-6510-4386-befc-ec0c5c031127-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.482937 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.482378 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8f2d86-6510-4386-befc-ec0c5c031127-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.482937 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.482631 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbcm\" (UniqueName: \"kubernetes.io/projected/8e8f2d86-6510-4386-befc-ec0c5c031127-kube-api-access-pjbcm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.483212 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.483192 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8f2d86-6510-4386-befc-ec0c5c031127-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.483426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.483404 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8e8f2d86-6510-4386-befc-ec0c5c031127-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.484944 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.484926 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8e8f2d86-6510-4386-befc-ec0c5c031127-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.496190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.496166 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbcm\" (UniqueName: \"kubernetes.io/projected/8e8f2d86-6510-4386-befc-ec0c5c031127-kube-api-access-pjbcm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-72fx4\" (UID: \"8e8f2d86-6510-4386-befc-ec0c5c031127\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.508990 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.508969 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:02.811298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:02.811265 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4"] Oct 31 03:03:02.811658 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:03:02.811630 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8f2d86_6510_4386_befc_ec0c5c031127.slice/crio-594b587122fcefee9edddc724c622b4fc405207147d60ac0eb36fc381dd3916f WatchSource:0}: Error finding container 594b587122fcefee9edddc724c622b4fc405207147d60ac0eb36fc381dd3916f: Status 404 returned error can't find the container with id 594b587122fcefee9edddc724c622b4fc405207147d60ac0eb36fc381dd3916f Oct 31 03:03:03.470497 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:03.470459 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" event={"ID":"8e8f2d86-6510-4386-befc-ec0c5c031127","Type":"ContainerStarted","Data":"594b587122fcefee9edddc724c622b4fc405207147d60ac0eb36fc381dd3916f"} Oct 31 03:03:04.289400 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:04.289359 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:03:04.292139 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:04.292116 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qprkr" Oct 31 03:03:12.465648 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.465615 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:03:12.466057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.465698 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:03:12.525780 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.524546 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bnt72"] Oct 31 03:03:12.537727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.537492 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-bnt72" podUID="4ea4d920-134e-44f9-9308-cf163c01b7b3" containerName="manager" containerID="cri-o://70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8" gracePeriod=30 Oct 31 03:03:12.823119 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.823086 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:03:12.875546 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.875516 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea4d920-134e-44f9-9308-cf163c01b7b3-kube-api-access-qnntt\" (UniqueName: \"kubernetes.io/projected/4ea4d920-134e-44f9-9308-cf163c01b7b3-kube-api-access-qnntt\") pod \"4ea4d920-134e-44f9-9308-cf163c01b7b3\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " Oct 31 03:03:12.875751 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.875725 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-metrics-certs\") pod \"4ea4d920-134e-44f9-9308-cf163c01b7b3\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " Oct 31 03:03:12.875834 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.875817 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-webhook-cert\") pod \"4ea4d920-134e-44f9-9308-cf163c01b7b3\" (UID: \"4ea4d920-134e-44f9-9308-cf163c01b7b3\") " Oct 31 03:03:12.879064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.878196 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea4d920-134e-44f9-9308-cf163c01b7b3-kube-api-access-qnntt" pod "4ea4d920-134e-44f9-9308-cf163c01b7b3" (UID: "4ea4d920-134e-44f9-9308-cf163c01b7b3"). InnerVolumeSpecName "kube-api-access-qnntt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:03:12.879064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.878209 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-webhook-cert" pod "4ea4d920-134e-44f9-9308-cf163c01b7b3" (UID: "4ea4d920-134e-44f9-9308-cf163c01b7b3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:03:12.879064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.878295 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-metrics-certs" pod "4ea4d920-134e-44f9-9308-cf163c01b7b3" (UID: "4ea4d920-134e-44f9-9308-cf163c01b7b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:03:12.977079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.976989 4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-metrics-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:12.977079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.977020 4456 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ea4d920-134e-44f9-9308-cf163c01b7b3-webhook-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:12.977079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:12.977047 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnntt\" (UniqueName: \"kubernetes.io/projected/4ea4d920-134e-44f9-9308-cf163c01b7b3-kube-api-access-qnntt\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:13.545702 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:13.545662 4456 generic.go:356] "Generic (PLEG): container finished" podID="4ea4d920-134e-44f9-9308-cf163c01b7b3" containerID="70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8" exitCode=0 Oct 31 03:03:13.546247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:13.545712 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-bnt72" Oct 31 03:03:13.546247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:13.545714 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-bnt72" event={"ID":"4ea4d920-134e-44f9-9308-cf163c01b7b3","Type":"ContainerDied","Data":"70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8"} Oct 31 03:03:13.546247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:13.545782 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-bnt72" event={"ID":"4ea4d920-134e-44f9-9308-cf163c01b7b3","Type":"ContainerDied","Data":"6dc579a200c08b4a90196e6bb1e8096c23fbb477a579e919fb4e2f8dd1cb7c95"} Oct 31 03:03:13.546247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:13.545817 4456 scope.go:122] "RemoveContainer" containerID="70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8" Oct 31 03:03:13.557806 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:13.557784 4456 scope.go:122] "RemoveContainer" containerID="70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8" Oct 31 03:03:13.558190 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:03:13.558152 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8\": container with ID starting with 70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8 not found: ID does not exist" containerID="70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8" Oct 31 03:03:13.558299 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:13.558202 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8"} err="failed to get container status \"70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8\": rpc error: code = NotFound desc = could not find container \"70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8\": container with ID starting with 70d276f2732f035976b080c50427ba3ddd48bdce4798784d3f9100ffe6cb0be8 not found: ID does not exist" Oct 31 03:03:13.573007 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:13.572957 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bnt72"] Oct 31 03:03:13.576839 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:13.576818 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bnt72"] Oct 31 03:03:14.440172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:14.440139 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea4d920-134e-44f9-9308-cf163c01b7b3" path="/var/lib/kubelet/pods/4ea4d920-134e-44f9-9308-cf163c01b7b3/volumes" Oct 31 03:03:14.553941 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:14.553905 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" event={"ID":"8e8f2d86-6510-4386-befc-ec0c5c031127","Type":"ContainerStarted","Data":"b04a01e24e58f08db960009ec0f626b4018d2ae76d23a0a36e98cf8f5b1c9d29"} Oct 31 03:03:14.554370 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:14.554060 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:14.579399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:14.579347 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" podStartSLOduration=1.533719639 podStartE2EDuration="12.579333482s" totalImagesPullingTime="11.045613843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:03:14.577662469 +0000 UTC m=+476.258188597" watchObservedRunningTime="2026-10-31 03:03:14.579333482 +0000 UTC m=+476.259859611" Oct 31 03:03:25.565440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:25.565408 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-72fx4" Oct 31 03:03:31.276200 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.276122 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj"] Oct 31 03:03:31.276686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.276667 4456 cpu_manager.go:414] "containerMap: removing container" podUID="4ea4d920-134e-44f9-9308-cf163c01b7b3" containerName="manager" Oct 31 03:03:31.276732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.276688 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea4d920-134e-44f9-9308-cf163c01b7b3" containerName="manager" Oct 31 03:03:31.276784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.276775 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea4d920-134e-44f9-9308-cf163c01b7b3" containerName="manager" Oct 31 03:03:31.281502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.281478 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.298991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.298947 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 03:03:31.299168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.299003 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vsq4\"" Oct 31 03:03:31.300563 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.300537 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj"] Oct 31 03:03:31.301072 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.301026 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:03:31.339894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.339855 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-metrics-certs\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.340109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.339907 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkxn\" (UniqueName: \"kubernetes.io/projected/54b40536-f284-42b3-97d8-c45d377ec542-kube-api-access-dlkxn\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.340109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.339983 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b40536-f284-42b3-97d8-c45d377ec542-openshift-service-ca\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.340254 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.340138 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-maas-api-tls\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.441287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.441233 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-maas-api-tls\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.441482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.441337 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-metrics-certs\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.441482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.441376 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkxn\" (UniqueName: \"kubernetes.io/projected/54b40536-f284-42b3-97d8-c45d377ec542-kube-api-access-dlkxn\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.441482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.441408 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b40536-f284-42b3-97d8-c45d377ec542-openshift-service-ca\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.442140 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.442117 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b40536-f284-42b3-97d8-c45d377ec542-openshift-service-ca\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.443912 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.443887 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-metrics-certs\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.443996 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.443938 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-maas-api-tls\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.461168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.461143 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkxn\" (UniqueName: \"kubernetes.io/projected/54b40536-f284-42b3-97d8-c45d377ec542-kube-api-access-dlkxn\") pod \"maas-api-6d669c7fcd-l68gj\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.594568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.594493 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:31.879291 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:31.879263 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj"] Oct 31 03:03:31.880456 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:03:31.880424 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b40536_f284_42b3_97d8_c45d377ec542.slice/crio-a77325b16f8d9991564488a8b917e9d5d135e6b035a5bfd1ae64e02abddc2632 WatchSource:0}: Error finding container a77325b16f8d9991564488a8b917e9d5d135e6b035a5bfd1ae64e02abddc2632: Status 404 returned error can't find the container with id a77325b16f8d9991564488a8b917e9d5d135e6b035a5bfd1ae64e02abddc2632 Oct 31 03:03:32.133051 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:03:32.132948 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:03:32.659701 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:32.659660 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" event={"ID":"54b40536-f284-42b3-97d8-c45d377ec542","Type":"ContainerStarted","Data":"a77325b16f8d9991564488a8b917e9d5d135e6b035a5bfd1ae64e02abddc2632"} Oct 31 03:03:34.683252 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:34.683216 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" event={"ID":"54b40536-f284-42b3-97d8-c45d377ec542","Type":"ContainerStarted","Data":"a450ed25e119fba3b114b2e12e66fb578ab2c8c384eef99b830944c9902ac163"} Oct 31 03:03:34.683775 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:34.683349 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:34.710013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:34.709967 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" podStartSLOduration=1.750291733 podStartE2EDuration="3.709956182s" totalImagesPullingTime="1.959664449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:03:34.708675575 +0000 UTC m=+496.389201702" watchObservedRunningTime="2026-10-31 03:03:34.709956182 +0000 UTC m=+496.390482352" Oct 31 03:03:40.694588 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:40.694558 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:42.730058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:42.730010 4456 generic.go:356] "Generic (PLEG): container finished" podID="ef33ec8e-50ff-4f0b-a84c-25a9b07c2221" containerID="bf7c53e1e5a5fd82a6c3f25fbe9d5056e5c7a4eb841ec7592388edfe57ecdd5f" exitCode=0 Oct 31 03:03:42.730578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:42.730086 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" event={"ID":"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221","Type":"ContainerDied","Data":"bf7c53e1e5a5fd82a6c3f25fbe9d5056e5c7a4eb841ec7592388edfe57ecdd5f"} Oct 31 03:03:42.730578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:42.730450 4456 scope.go:122] "RemoveContainer" containerID="bf7c53e1e5a5fd82a6c3f25fbe9d5056e5c7a4eb841ec7592388edfe57ecdd5f" Oct 31 03:03:42.731563 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:42.731527 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:03:42.731660 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:42.731614 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:03:42.731719 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:42.731667 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:03:43.737456 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:43.737419 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" event={"ID":"ef33ec8e-50ff-4f0b-a84c-25a9b07c2221","Type":"ContainerStarted","Data":"ba5a364574fa8fd971f6afafa26474adc251f2207803a20bd14d66d489122a80"} Oct 31 03:03:44.510818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:44.510780 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:03:44.511563 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:44.511542 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7r8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:03:44.511621 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:44.511588 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" podUID="ef33ec8e-50ff-4f0b-a84c-25a9b07c2221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 03:03:44.966061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:44.953137 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:03:45.107719 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.107677 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp"] Oct 31 03:03:45.163284 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.163250 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp"] Oct 31 03:03:45.163602 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.163584 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" Oct 31 03:03:45.286662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.286561 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af4ef643-fdf2-41ab-ae84-82c812b965e6-cert\") pod \"llmisvc-controller-manager-6466c554d8-wwfhp\" (UID: \"af4ef643-fdf2-41ab-ae84-82c812b965e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" Oct 31 03:03:45.286662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.286632 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz465\" (UniqueName: \"kubernetes.io/projected/af4ef643-fdf2-41ab-ae84-82c812b965e6-kube-api-access-dz465\") pod \"llmisvc-controller-manager-6466c554d8-wwfhp\" (UID: \"af4ef643-fdf2-41ab-ae84-82c812b965e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" Oct 31 03:03:45.387225 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.387187 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz465\" (UniqueName: \"kubernetes.io/projected/af4ef643-fdf2-41ab-ae84-82c812b965e6-kube-api-access-dz465\") pod \"llmisvc-controller-manager-6466c554d8-wwfhp\" (UID: \"af4ef643-fdf2-41ab-ae84-82c812b965e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" Oct 31 03:03:45.387419 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.387266 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af4ef643-fdf2-41ab-ae84-82c812b965e6-cert\") pod \"llmisvc-controller-manager-6466c554d8-wwfhp\" (UID: \"af4ef643-fdf2-41ab-ae84-82c812b965e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" Oct 31 03:03:45.389802 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.389776 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af4ef643-fdf2-41ab-ae84-82c812b965e6-cert\") pod \"llmisvc-controller-manager-6466c554d8-wwfhp\" (UID: \"af4ef643-fdf2-41ab-ae84-82c812b965e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" Oct 31 03:03:45.401130 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.401105 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz465\" (UniqueName: \"kubernetes.io/projected/af4ef643-fdf2-41ab-ae84-82c812b965e6-kube-api-access-dz465\") pod \"llmisvc-controller-manager-6466c554d8-wwfhp\" (UID: \"af4ef643-fdf2-41ab-ae84-82c812b965e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" Oct 31 03:03:45.498113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.498078 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" Oct 31 03:03:45.511492 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.511465 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7r8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:03:45.511597 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.511514 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" podUID="ef33ec8e-50ff-4f0b-a84c-25a9b07c2221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 03:03:45.841945 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:45.841916 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp"] Oct 31 03:03:45.842369 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:03:45.842342 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4ef643_fdf2_41ab_ae84_82c812b965e6.slice/crio-dccc857e57b0c3f0334ba14ddd3f8193f8c19dda9ca0ad799c7663b77b2025ff WatchSource:0}: Error finding container dccc857e57b0c3f0334ba14ddd3f8193f8c19dda9ca0ad799c7663b77b2025ff: Status 404 returned error can't find the container with id dccc857e57b0c3f0334ba14ddd3f8193f8c19dda9ca0ad799c7663b77b2025ff Oct 31 03:03:46.510878 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:46.510841 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7r8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:03:46.511380 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:46.510899 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" podUID="ef33ec8e-50ff-4f0b-a84c-25a9b07c2221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 03:03:46.783818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:46.783787 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" event={"ID":"af4ef643-fdf2-41ab-ae84-82c812b965e6","Type":"ContainerStarted","Data":"4f85799e508e22b9843289b9022f9d3fb8871dd5ba5ce4995b35f311718b00e9"} Oct 31 03:03:46.783946 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:46.783845 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" event={"ID":"af4ef643-fdf2-41ab-ae84-82c812b965e6","Type":"ContainerStarted","Data":"dccc857e57b0c3f0334ba14ddd3f8193f8c19dda9ca0ad799c7663b77b2025ff"} Oct 31 03:03:47.511060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:47.510948 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7r8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:03:47.511060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:47.510999 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" podUID="ef33ec8e-50ff-4f0b-a84c-25a9b07c2221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 03:03:47.790066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:47.789961 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" Oct 31 03:03:47.812282 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:47.812230 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" podStartSLOduration=2.010656611 podStartE2EDuration="2.812216794s" totalImagesPullingTime="801.560183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:03:47.810814025 +0000 UTC m=+509.491340151" watchObservedRunningTime="2026-10-31 03:03:47.812216794 +0000 UTC m=+509.492742932" Oct 31 03:03:48.510609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:48.510567 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7r8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:03:48.510798 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:48.510625 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" podUID="ef33ec8e-50ff-4f0b-a84c-25a9b07c2221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 03:03:49.514852 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:49.514819 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:03:49.515795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:49.515774 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" Oct 31 03:03:51.647451 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.647421 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj"] Oct 31 03:03:51.647883 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.647615 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" podUID="54b40536-f284-42b3-97d8-c45d377ec542" containerName="maas-api" containerID="cri-o://a450ed25e119fba3b114b2e12e66fb578ab2c8c384eef99b830944c9902ac163" gracePeriod=30 Oct 31 03:03:51.818562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.818290 4456 generic.go:356] "Generic (PLEG): container finished" podID="54b40536-f284-42b3-97d8-c45d377ec542" containerID="a450ed25e119fba3b114b2e12e66fb578ab2c8c384eef99b830944c9902ac163" exitCode=0 Oct 31 03:03:51.818562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.818397 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" event={"ID":"54b40536-f284-42b3-97d8-c45d377ec542","Type":"ContainerDied","Data":"a450ed25e119fba3b114b2e12e66fb578ab2c8c384eef99b830944c9902ac163"} Oct 31 03:03:51.890233 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.890206 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:51.944260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.944226 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-metrics-certs\") pod \"54b40536-f284-42b3-97d8-c45d377ec542\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " Oct 31 03:03:51.944412 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.944285 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-maas-api-tls\") pod \"54b40536-f284-42b3-97d8-c45d377ec542\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " Oct 31 03:03:51.944412 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.944349 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b40536-f284-42b3-97d8-c45d377ec542-kube-api-access-dlkxn\" (UniqueName: \"kubernetes.io/projected/54b40536-f284-42b3-97d8-c45d377ec542-kube-api-access-dlkxn\") pod \"54b40536-f284-42b3-97d8-c45d377ec542\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " Oct 31 03:03:51.944412 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.944368 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b40536-f284-42b3-97d8-c45d377ec542-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b40536-f284-42b3-97d8-c45d377ec542-openshift-service-ca\") pod \"54b40536-f284-42b3-97d8-c45d377ec542\" (UID: \"54b40536-f284-42b3-97d8-c45d377ec542\") " Oct 31 03:03:51.944853 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.944818 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b40536-f284-42b3-97d8-c45d377ec542-openshift-service-ca" pod "54b40536-f284-42b3-97d8-c45d377ec542" (UID: "54b40536-f284-42b3-97d8-c45d377ec542"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:03:51.946458 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.946428 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b40536-f284-42b3-97d8-c45d377ec542-kube-api-access-dlkxn" pod "54b40536-f284-42b3-97d8-c45d377ec542" (UID: "54b40536-f284-42b3-97d8-c45d377ec542"). InnerVolumeSpecName "kube-api-access-dlkxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:03:51.946600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.946468 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-metrics-certs" pod "54b40536-f284-42b3-97d8-c45d377ec542" (UID: "54b40536-f284-42b3-97d8-c45d377ec542"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:03:51.946600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:51.946531 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-maas-api-tls" pod "54b40536-f284-42b3-97d8-c45d377ec542" (UID: "54b40536-f284-42b3-97d8-c45d377ec542"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:03:52.046049 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:52.046006 4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-metrics-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:52.046049 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:52.046060 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b40536-f284-42b3-97d8-c45d377ec542-maas-api-tls\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:52.046244 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:52.046072 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlkxn\" (UniqueName: \"kubernetes.io/projected/54b40536-f284-42b3-97d8-c45d377ec542-kube-api-access-dlkxn\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:52.046244 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:52.046083 4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b40536-f284-42b3-97d8-c45d377ec542-openshift-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:52.824813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:52.824730 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" event={"ID":"54b40536-f284-42b3-97d8-c45d377ec542","Type":"ContainerDied","Data":"a77325b16f8d9991564488a8b917e9d5d135e6b035a5bfd1ae64e02abddc2632"} Oct 31 03:03:52.824813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:52.824767 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj" Oct 31 03:03:52.824813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:52.824793 4456 scope.go:122] "RemoveContainer" containerID="a450ed25e119fba3b114b2e12e66fb578ab2c8c384eef99b830944c9902ac163" Oct 31 03:03:52.846453 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:52.846420 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj"] Oct 31 03:03:52.850885 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:52.850859 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-l68gj"] Oct 31 03:03:54.435829 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:03:54.435800 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b40536-f284-42b3-97d8-c45d377ec542" path="/var/lib/kubelet/pods/54b40536-f284-42b3-97d8-c45d377ec542/volumes" Oct 31 03:04:02.133127 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:04:02.133091 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:04:18.799527 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:18.799443 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wwfhp" Oct 31 03:04:18.853344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:18.853303 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs"] Oct 31 03:04:18.853585 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:18.853553 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" podUID="c1b2a81a-0acd-43cb-b216-49b14b24b450" containerName="manager" containerID="cri-o://d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1" gracePeriod=30 Oct 31 03:04:19.096106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.096082 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:04:19.193256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.193223 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1b2a81a-0acd-43cb-b216-49b14b24b450-cert\" (UniqueName: \"kubernetes.io/secret/c1b2a81a-0acd-43cb-b216-49b14b24b450-cert\") pod \"c1b2a81a-0acd-43cb-b216-49b14b24b450\" (UID: \"c1b2a81a-0acd-43cb-b216-49b14b24b450\") " Oct 31 03:04:19.193416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.193271 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b2a81a-0acd-43cb-b216-49b14b24b450-kube-api-access-4s4l5\" (UniqueName: \"kubernetes.io/projected/c1b2a81a-0acd-43cb-b216-49b14b24b450-kube-api-access-4s4l5\") pod \"c1b2a81a-0acd-43cb-b216-49b14b24b450\" (UID: \"c1b2a81a-0acd-43cb-b216-49b14b24b450\") " Oct 31 03:04:19.195263 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.195231 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b2a81a-0acd-43cb-b216-49b14b24b450-cert" pod "c1b2a81a-0acd-43cb-b216-49b14b24b450" (UID: "c1b2a81a-0acd-43cb-b216-49b14b24b450"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:04:19.195372 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.195306 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b2a81a-0acd-43cb-b216-49b14b24b450-kube-api-access-4s4l5" pod "c1b2a81a-0acd-43cb-b216-49b14b24b450" (UID: "c1b2a81a-0acd-43cb-b216-49b14b24b450"). InnerVolumeSpecName "kube-api-access-4s4l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:04:19.294401 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.294368 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s4l5\" (UniqueName: \"kubernetes.io/projected/c1b2a81a-0acd-43cb-b216-49b14b24b450-kube-api-access-4s4l5\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:04:19.294401 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.294395 4456 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1b2a81a-0acd-43cb-b216-49b14b24b450-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:04:19.973695 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.973663 4456 generic.go:356] "Generic (PLEG): container finished" podID="c1b2a81a-0acd-43cb-b216-49b14b24b450" containerID="d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1" exitCode=0 Oct 31 03:04:19.974185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.973709 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" Oct 31 03:04:19.974185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.973750 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" event={"ID":"c1b2a81a-0acd-43cb-b216-49b14b24b450","Type":"ContainerDied","Data":"d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1"} Oct 31 03:04:19.974185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.973809 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs" event={"ID":"c1b2a81a-0acd-43cb-b216-49b14b24b450","Type":"ContainerDied","Data":"a9d26cb4a1c6d63e3a3073457cd4d6c3248460ab632ca1f31df7a83654b01ff3"} Oct 31 03:04:19.974185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.973841 4456 scope.go:122] "RemoveContainer" containerID="d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1" Oct 31 03:04:19.983578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.983556 4456 scope.go:122] "RemoveContainer" containerID="d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1" Oct 31 03:04:19.983823 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:04:19.983806 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1\": container with ID starting with d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1 not found: ID does not exist" containerID="d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1" Oct 31 03:04:19.983879 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.983832 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1"} err="failed to get container status \"d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1\": rpc error: code = NotFound desc = could not find container \"d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1\": container with ID starting with d4697150f2fe460e58b41b5ea65a38c15ad978e97f5d1863b80b9088291378e1 not found: ID does not exist" Oct 31 03:04:19.996764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:19.996742 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs"] Oct 31 03:04:20.001499 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:20.001478 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s5xvs"] Oct 31 03:04:20.434293 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:04:20.434264 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b2a81a-0acd-43cb-b216-49b14b24b450" path="/var/lib/kubelet/pods/c1b2a81a-0acd-43cb-b216-49b14b24b450/volumes" Oct 31 03:04:32.132331 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:04:32.132293 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:05:02.133523 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:05:02.133432 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:05:16.705408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.705377 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw"] Oct 31 03:05:16.706020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.705993 4456 cpu_manager.go:414] "containerMap: removing container" podUID="54b40536-f284-42b3-97d8-c45d377ec542" containerName="maas-api" Oct 31 03:05:16.706020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.706019 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b40536-f284-42b3-97d8-c45d377ec542" containerName="maas-api" Oct 31 03:05:16.706196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.706050 4456 cpu_manager.go:414] "containerMap: removing container" podUID="c1b2a81a-0acd-43cb-b216-49b14b24b450" containerName="manager" Oct 31 03:05:16.706196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.706059 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b2a81a-0acd-43cb-b216-49b14b24b450" containerName="manager" Oct 31 03:05:16.706196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.706176 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b2a81a-0acd-43cb-b216-49b14b24b450" containerName="manager" Oct 31 03:05:16.706308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.706201 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b40536-f284-42b3-97d8-c45d377ec542" containerName="maas-api" Oct 31 03:05:16.709553 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.709534 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.720725 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.720695 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw"] Oct 31 03:05:16.810336 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.810251 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.810336 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.810289 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.810336 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.810312 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjg8k\" (UniqueName: \"kubernetes.io/projected/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-kube-api-access-tjg8k\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.810546 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.810428 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.810546 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.810471 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.810546 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.810514 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.810641 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.810563 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.810641 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.810605 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.810641 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.810623 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911022 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.910990 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911022 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911024 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911077 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjg8k\" (UniqueName: \"kubernetes.io/projected/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-kube-api-access-tjg8k\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911103 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911128 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911182 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911218 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911271 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911295 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911527 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911627 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911808 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911783 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911783 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.911882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.911790 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.913522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.913499 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.913745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.913728 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.924490 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.924466 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:16.924671 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:16.924653 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjg8k\" (UniqueName: \"kubernetes.io/projected/4f1f2ef9-d74d-4c50-8eeb-60103414adaa-kube-api-access-tjg8k\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gdw\" (UID: \"4f1f2ef9-d74d-4c50-8eeb-60103414adaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:17.029197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:17.029168 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:17.309509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:17.309481 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw"] Oct 31 03:05:17.310030 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:05:17.309998 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1f2ef9_d74d_4c50_8eeb_60103414adaa.slice/crio-fe4e614cb2a16bdf393b5b0c7606bc242b7ae09f924181c92d2c4cfa3c171e94 WatchSource:0}: Error finding container fe4e614cb2a16bdf393b5b0c7606bc242b7ae09f924181c92d2c4cfa3c171e94: Status 404 returned error can't find the container with id fe4e614cb2a16bdf393b5b0c7606bc242b7ae09f924181c92d2c4cfa3c171e94 Oct 31 03:05:17.312205 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:17.312166 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:05:17.312304 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:17.312274 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:05:17.312369 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:17.312320 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:05:17.319201 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:17.319179 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" event={"ID":"4f1f2ef9-d74d-4c50-8eeb-60103414adaa","Type":"ContainerStarted","Data":"fe4e614cb2a16bdf393b5b0c7606bc242b7ae09f924181c92d2c4cfa3c171e94"} Oct 31 03:05:18.327243 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:18.327205 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" event={"ID":"4f1f2ef9-d74d-4c50-8eeb-60103414adaa","Type":"ContainerStarted","Data":"af25acc5287f855129ece0e3df7c37af5c34e5edffecc0ed9dba9095aaeea76a"} Oct 31 03:05:19.029540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:19.029502 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:19.030931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:19.030904 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:05:19.031067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:19.030962 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" podUID="4f1f2ef9-d74d-4c50-8eeb-60103414adaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 03:05:20.029890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:20.029851 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:05:20.030342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:20.029909 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" podUID="4f1f2ef9-d74d-4c50-8eeb-60103414adaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 03:05:21.030007 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:21.029973 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:05:21.030406 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:21.030022 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" podUID="4f1f2ef9-d74d-4c50-8eeb-60103414adaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 03:05:22.029757 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:22.029721 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:05:22.029916 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:22.029777 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" podUID="4f1f2ef9-d74d-4c50-8eeb-60103414adaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 03:05:23.030221 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:23.030183 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:05:23.030628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:23.030242 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" podUID="4f1f2ef9-d74d-4c50-8eeb-60103414adaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 03:05:24.030345 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:24.030301 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:05:24.030770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:24.030394 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" podUID="4f1f2ef9-d74d-4c50-8eeb-60103414adaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 03:05:25.437679 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:25.437644 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:25.438209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:25.438027 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:25.438762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:25.438738 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" Oct 31 03:05:25.467579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:05:25.467534 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" podStartSLOduration=9.467523169 podStartE2EDuration="9.467523169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:18.37209106 +0000 UTC m=+600.052617188" watchObservedRunningTime="2026-10-31 03:05:25.467523169 +0000 UTC m=+607.148049295" Oct 31 03:05:32.134096 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:05:32.134058 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:06:02.133209 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:06:02.133173 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:06:18.935426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:18.935385 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m"] Oct 31 03:06:18.939120 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:18.939094 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:18.950747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:18.950341 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-56afbd-serving-cert\"" Oct 31 03:06:18.950747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:18.950603 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vsq4\"" Oct 31 03:06:18.953708 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:18.950684 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:06:18.953708 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:18.953537 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m"] Oct 31 03:06:18.999111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:18.999079 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821d763a-861c-4d02-96b9-39aa69554d1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:18.999297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:18.999120 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmklp\" (UniqueName: \"kubernetes.io/projected/821d763a-861c-4d02-96b9-39aa69554d1a-kube-api-access-nmklp\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:18.999297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:18.999185 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-metrics-certs\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:18.999419 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:18.999323 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:19.100238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.100205 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821d763a-861c-4d02-96b9-39aa69554d1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:19.100434 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.100263 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmklp\" (UniqueName: \"kubernetes.io/projected/821d763a-861c-4d02-96b9-39aa69554d1a-kube-api-access-nmklp\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:19.100434 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.100322 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-metrics-certs\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:19.100434 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.100390 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:19.101078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.101051 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821d763a-861c-4d02-96b9-39aa69554d1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:19.103073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.103049 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-metrics-certs\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:19.103172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.103131 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:19.116190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.116084 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmklp\" (UniqueName: \"kubernetes.io/projected/821d763a-861c-4d02-96b9-39aa69554d1a-kube-api-access-nmklp\") pod \"maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:19.256155 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.256070 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:19.544776 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.544690 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m"] Oct 31 03:06:19.550235 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:06:19.550187 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821d763a_861c_4d02_96b9_39aa69554d1a.slice/crio-c19f620bd5c60a0dbef44e7c67522c2140b930b0e4720469353443cabb43a04e WatchSource:0}: Error finding container c19f620bd5c60a0dbef44e7c67522c2140b930b0e4720469353443cabb43a04e: Status 404 returned error can't find the container with id c19f620bd5c60a0dbef44e7c67522c2140b930b0e4720469353443cabb43a04e Oct 31 03:06:19.554032 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.554007 4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:06:19.688755 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:19.688714 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" event={"ID":"821d763a-861c-4d02-96b9-39aa69554d1a","Type":"ContainerStarted","Data":"c19f620bd5c60a0dbef44e7c67522c2140b930b0e4720469353443cabb43a04e"} Oct 31 03:06:20.198311 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.198279 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g"] Oct 31 03:06:20.202885 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.202860 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.216174 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.216148 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g"] Oct 31 03:06:20.302129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.302022 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-85cb59-serving-cert\"" Oct 31 03:06:20.314569 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.314533 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-maas-api-tls\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.314763 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.314588 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-metrics-certs\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.314763 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.314621 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszf2\" (UniqueName: \"kubernetes.io/projected/90490dc2-8174-4f93-b613-db807dd23c34-kube-api-access-qszf2\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.314763 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.314678 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90490dc2-8174-4f93-b613-db807dd23c34-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.416103 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.416056 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90490dc2-8174-4f93-b613-db807dd23c34-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.416302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.416157 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-maas-api-tls\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.416302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.416190 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-metrics-certs\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.416302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.416232 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszf2\" (UniqueName: \"kubernetes.io/projected/90490dc2-8174-4f93-b613-db807dd23c34-kube-api-access-qszf2\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.416480 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:06:20.416314 4456 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-serving-cert: secret "maas-api-e2e-worker-w2-85cb59-serving-cert" not found Oct 31 03:06:20.416480 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:06:20.416385 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-maas-api-tls podName:90490dc2-8174-4f93-b613-db807dd23c34 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:20.916367593 +0000 UTC m=+662.596893711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-maas-api-tls") pod "maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" (UID: "90490dc2-8174-4f93-b613-db807dd23c34") : secret "maas-api-e2e-worker-w2-85cb59-serving-cert" not found Oct 31 03:06:20.416708 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.416685 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90490dc2-8174-4f93-b613-db807dd23c34-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.418598 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.418570 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-metrics-certs\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.437606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.437574 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszf2\" (UniqueName: \"kubernetes.io/projected/90490dc2-8174-4f93-b613-db807dd23c34-kube-api-access-qszf2\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.698760 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.698592 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" event={"ID":"821d763a-861c-4d02-96b9-39aa69554d1a","Type":"ContainerStarted","Data":"0a496eb301e00234bc3e7fd51bdef0dd4c063fc34570f4ce375f5386b8c12b0c"} Oct 31 03:06:20.698760 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.698671 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:20.723871 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.723822 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" podStartSLOduration=2.227527761 podStartE2EDuration="2.7238063s" totalImagesPullingTime="496.278539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:20.722108163 +0000 UTC m=+662.402634291" watchObservedRunningTime="2026-10-31 03:06:20.7238063 +0000 UTC m=+662.404332429" Oct 31 03:06:20.921489 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.921451 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-maas-api-tls\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:20.927663 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:20.927630 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-maas-api-tls\") pod \"maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:21.122218 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:21.122109 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:21.408865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:21.408831 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g"] Oct 31 03:06:21.410134 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:06:21.410094 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90490dc2_8174_4f93_b613_db807dd23c34.slice/crio-f7e28de381d5937c4267a65b364dec338286e3bac445ab9c538f1aa862c07b88 WatchSource:0}: Error finding container f7e28de381d5937c4267a65b364dec338286e3bac445ab9c538f1aa862c07b88: Status 404 returned error can't find the container with id f7e28de381d5937c4267a65b364dec338286e3bac445ab9c538f1aa862c07b88 Oct 31 03:06:21.706117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:21.706016 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" event={"ID":"90490dc2-8174-4f93-b613-db807dd23c34","Type":"ContainerStarted","Data":"f7e28de381d5937c4267a65b364dec338286e3bac445ab9c538f1aa862c07b88"} Oct 31 03:06:22.715346 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:22.715308 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" event={"ID":"90490dc2-8174-4f93-b613-db807dd23c34","Type":"ContainerStarted","Data":"e459d5521a99b8349e2bd684bca8af35a89a1f88326ab56688d2876c4fd42069"} Oct 31 03:06:22.715817 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:22.715379 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:22.738960 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:22.738914 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" podStartSLOduration=2.31592067 podStartE2EDuration="2.738899453s" totalImagesPullingTime="422.978783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:22.736973984 +0000 UTC m=+664.417500111" watchObservedRunningTime="2026-10-31 03:06:22.738899453 +0000 UTC m=+664.419425580" Oct 31 03:06:23.033611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.033538 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj"] Oct 31 03:06:23.037194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.037161 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.047964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.047937 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-99bc97-serving-cert\"" Oct 31 03:06:23.049498 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.049465 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj"] Oct 31 03:06:23.143795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.143766 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-metrics-certs\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.143970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.143801 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-maas-api-tls\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.143970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.143841 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f08352-6eb7-40f5-8672-1de230b9d165-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.143970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.143944 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgrc\" (UniqueName: \"kubernetes.io/projected/90f08352-6eb7-40f5-8672-1de230b9d165-kube-api-access-4pgrc\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.254906 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.254854 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-metrics-certs\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.255142 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.254913 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-maas-api-tls\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.255142 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.254975 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f08352-6eb7-40f5-8672-1de230b9d165-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.255142 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.255016 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgrc\" (UniqueName: \"kubernetes.io/projected/90f08352-6eb7-40f5-8672-1de230b9d165-kube-api-access-4pgrc\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.256534 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.256487 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f08352-6eb7-40f5-8672-1de230b9d165-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.257796 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.257771 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-metrics-certs\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.258021 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.257989 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-maas-api-tls\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.268903 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.268870 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgrc\" (UniqueName: \"kubernetes.io/projected/90f08352-6eb7-40f5-8672-1de230b9d165-kube-api-access-4pgrc\") pod \"maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.357947 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.357847 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:23.640325 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.640299 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj"] Oct 31 03:06:23.641002 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:06:23.640973 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f08352_6eb7_40f5_8672_1de230b9d165.slice/crio-329338c1cd7343155113dfe2ca9d9c5816ce74a3d89bdd13d59974c5cb69a325 WatchSource:0}: Error finding container 329338c1cd7343155113dfe2ca9d9c5816ce74a3d89bdd13d59974c5cb69a325: Status 404 returned error can't find the container with id 329338c1cd7343155113dfe2ca9d9c5816ce74a3d89bdd13d59974c5cb69a325 Oct 31 03:06:23.722737 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.722699 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" event={"ID":"90f08352-6eb7-40f5-8672-1de230b9d165","Type":"ContainerStarted","Data":"329338c1cd7343155113dfe2ca9d9c5816ce74a3d89bdd13d59974c5cb69a325"} Oct 31 03:06:23.939632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.939599 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s"] Oct 31 03:06:23.945226 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.945191 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:23.955070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.954259 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-edb9e6-serving-cert\"" Oct 31 03:06:23.955070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:23.954351 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s"] Oct 31 03:06:24.063152 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.063116 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7666a98d-9d6a-439f-a141-771378ad2861-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.063278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.063165 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-maas-api-tls\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.063349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.063265 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf4rt\" (UniqueName: \"kubernetes.io/projected/7666a98d-9d6a-439f-a141-771378ad2861-kube-api-access-qf4rt\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.063349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.063323 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-metrics-certs\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.164343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.164298 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7666a98d-9d6a-439f-a141-771378ad2861-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.164343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.164343 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-maas-api-tls\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.164569 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.164366 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf4rt\" (UniqueName: \"kubernetes.io/projected/7666a98d-9d6a-439f-a141-771378ad2861-kube-api-access-qf4rt\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.164569 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.164398 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-metrics-certs\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.165052 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.165011 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7666a98d-9d6a-439f-a141-771378ad2861-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.167078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.167053 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-maas-api-tls\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.167909 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.167876 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-metrics-certs\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.177690 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.177661 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf4rt\" (UniqueName: \"kubernetes.io/projected/7666a98d-9d6a-439f-a141-771378ad2861-kube-api-access-qf4rt\") pod \"maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.267894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.267794 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:24.552275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.552247 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s"] Oct 31 03:06:24.552938 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:06:24.552908 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7666a98d_9d6a_439f_a141_771378ad2861.slice/crio-600f21c48e85ede2d17f52fe2350762f00f8b2f6cdb07af1e36a4524da9f0976 WatchSource:0}: Error finding container 600f21c48e85ede2d17f52fe2350762f00f8b2f6cdb07af1e36a4524da9f0976: Status 404 returned error can't find the container with id 600f21c48e85ede2d17f52fe2350762f00f8b2f6cdb07af1e36a4524da9f0976 Oct 31 03:06:24.729976 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.729941 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" event={"ID":"7666a98d-9d6a-439f-a141-771378ad2861","Type":"ContainerStarted","Data":"600f21c48e85ede2d17f52fe2350762f00f8b2f6cdb07af1e36a4524da9f0976"} Oct 31 03:06:24.731498 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.731474 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" event={"ID":"90f08352-6eb7-40f5-8672-1de230b9d165","Type":"ContainerStarted","Data":"5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f"} Oct 31 03:06:24.731639 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.731590 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:24.759851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:24.759794 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" podStartSLOduration=1.352357343 podStartE2EDuration="1.759775627s" totalImagesPullingTime="407.418284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:24.757198263 +0000 UTC m=+666.437724390" watchObservedRunningTime="2026-10-31 03:06:24.759775627 +0000 UTC m=+666.440301755" Oct 31 03:06:25.738938 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:25.738901 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" event={"ID":"7666a98d-9d6a-439f-a141-771378ad2861","Type":"ContainerStarted","Data":"72e84b79749e1fb1ebc25a34eb25bfbff0b1b71cad3fcbc42c42dfac746a6896"} Oct 31 03:06:25.739386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:25.739150 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:25.790820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:25.790755 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" podStartSLOduration=2.298005425 podStartE2EDuration="2.790741571s" totalImagesPullingTime="492.736146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:25.788748484 +0000 UTC m=+667.469274613" watchObservedRunningTime="2026-10-31 03:06:25.790741571 +0000 UTC m=+667.471267716" Oct 31 03:06:26.711808 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:26.711776 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:06:28.729279 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:28.729248 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:06:30.744888 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:30.744855 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:06:31.752155 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:31.752122 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:06:32.134440 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:06:32.134347 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:06:50.454755 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.454722 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm"] Oct 31 03:06:50.458872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.458845 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.472463 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.472431 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bebea4\"/\"kube-root-ca.crt\"" Oct 31 03:06:50.472612 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.472546 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebea4\"/\"default-dockercfg-mgsmr\"" Oct 31 03:06:50.472681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.472619 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bebea4\"/\"openshift-service-ca.crt\"" Oct 31 03:06:50.472740 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.472719 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebea4\"/\"simulator-w4-bebea4-kserve-self-signed-certs\"" Oct 31 03:06:50.474711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.474684 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm"] Oct 31 03:06:50.535024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.534986 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-home\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.535024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.535024 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-tmp-dir\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.535251 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.535202 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-dshm\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.535288 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.535264 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-model-cache\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.535393 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.535376 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvmq\" (UniqueName: \"kubernetes.io/projected/391ef3a2-cff8-481b-898a-b00a84351d8d-kube-api-access-bmvmq\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.535457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.535443 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391ef3a2-cff8-481b-898a-b00a84351d8d-tls-certs\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.636234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.636198 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvmq\" (UniqueName: \"kubernetes.io/projected/391ef3a2-cff8-481b-898a-b00a84351d8d-kube-api-access-bmvmq\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.636418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.636242 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391ef3a2-cff8-481b-898a-b00a84351d8d-tls-certs\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.636418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.636295 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-home\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.636418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.636311 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-tmp-dir\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.636418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.636341 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-dshm\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.636418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.636378 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-model-cache\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.636747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.636726 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-tmp-dir\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.636819 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.636773 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-model-cache\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.636867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.636837 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-home\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.638723 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.638702 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-dshm\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.639006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.638981 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391ef3a2-cff8-481b-898a-b00a84351d8d-tls-certs\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.649833 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.649807 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvmq\" (UniqueName: \"kubernetes.io/projected/391ef3a2-cff8-481b-898a-b00a84351d8d-kube-api-access-bmvmq\") pod \"simulator-w4-bebea4-kserve-684c9bd584-gkbbm\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:50.779081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:50.778973 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:06:51.067087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:51.067054 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm"] Oct 31 03:06:51.068636 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:06:51.068608 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod391ef3a2_cff8_481b_898a_b00a84351d8d.slice/crio-5afdd26c17b82351d9911c92f6660a8ac5e66b9f43df7290c24cc3aa057a1e70 WatchSource:0}: Error finding container 5afdd26c17b82351d9911c92f6660a8ac5e66b9f43df7290c24cc3aa057a1e70: Status 404 returned error can't find the container with id 5afdd26c17b82351d9911c92f6660a8ac5e66b9f43df7290c24cc3aa057a1e70 Oct 31 03:06:51.922323 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:51.922282 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" event={"ID":"391ef3a2-cff8-481b-898a-b00a84351d8d","Type":"ContainerStarted","Data":"5afdd26c17b82351d9911c92f6660a8ac5e66b9f43df7290c24cc3aa057a1e70"} Oct 31 03:06:52.929502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:52.929466 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" event={"ID":"391ef3a2-cff8-481b-898a-b00a84351d8d","Type":"ContainerStarted","Data":"d3928741a641c1eddfc45e12fb9b443476af4aae996032df0ac5e24f2b705975"} Oct 31 03:06:52.951564 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:52.951513 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podStartSLOduration=1.387508964 podStartE2EDuration="2.951499136s" totalImagesPullingTime="1.563990172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:52.950684305 +0000 UTC m=+694.631210432" watchObservedRunningTime="2026-10-31 03:06:52.951499136 +0000 UTC m=+694.632025263" Oct 31 03:06:53.779741 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:06:53.779703 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:07:00.780001 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:00.779964 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:07:00.793765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:00.793729 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:07:00.992962 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:00.992928 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:07:02.132859 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:07:02.132823 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:07:02.451643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.451602 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv"] Oct 31 03:07:02.457393 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.457363 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.471652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.471626 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv"] Oct 31 03:07:02.472134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.472110 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"default-dockercfg-zjnj2\"" Oct 31 03:07:02.472266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.472229 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"kube-root-ca.crt\"" Oct 31 03:07:02.472335 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.472322 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"simulator-w6-7d8daf-kserve-self-signed-certs\"" Oct 31 03:07:02.472408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.472382 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"openshift-service-ca.crt\"" Oct 31 03:07:02.576778 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.576738 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-home\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.576964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.576849 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-dshm\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.576964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.576882 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-tmp-dir\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.577125 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.576955 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-model-cache\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.577182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.577135 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swfrn\" (UniqueName: \"kubernetes.io/projected/a0661629-6e95-40c9-a8d1-4646ae2634f9-kube-api-access-swfrn\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.577237 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.577205 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0661629-6e95-40c9-a8d1-4646ae2634f9-tls-certs\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.678296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.678263 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-tmp-dir\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.678532 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.678304 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-model-cache\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.678532 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.678334 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfrn\" (UniqueName: \"kubernetes.io/projected/a0661629-6e95-40c9-a8d1-4646ae2634f9-kube-api-access-swfrn\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.678532 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.678372 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0661629-6e95-40c9-a8d1-4646ae2634f9-tls-certs\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.678532 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.678417 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-home\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.678532 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.678465 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-dshm\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.678819 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.678767 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-model-cache\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.678819 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.678799 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-tmp-dir\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.678931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.678908 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-home\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.680642 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.680626 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-dshm\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.680908 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.680893 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0661629-6e95-40c9-a8d1-4646ae2634f9-tls-certs\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.692003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.691976 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfrn\" (UniqueName: \"kubernetes.io/projected/a0661629-6e95-40c9-a8d1-4646ae2634f9-kube-api-access-swfrn\") pod \"simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:02.771940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:02.771833 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:03.055354 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:07:03.055323 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0661629_6e95_40c9_a8d1_4646ae2634f9.slice/crio-b00ac30a17e71e5f5c4cb5e59ca17acdc9aaf8595949fce8dfbbdc44ef20d2d9 WatchSource:0}: Error finding container b00ac30a17e71e5f5c4cb5e59ca17acdc9aaf8595949fce8dfbbdc44ef20d2d9: Status 404 returned error can't find the container with id b00ac30a17e71e5f5c4cb5e59ca17acdc9aaf8595949fce8dfbbdc44ef20d2d9 Oct 31 03:07:03.056145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:03.056111 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv"] Oct 31 03:07:04.002253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:04.002215 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" event={"ID":"a0661629-6e95-40c9-a8d1-4646ae2634f9","Type":"ContainerStarted","Data":"e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff"} Oct 31 03:07:04.002659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:04.002264 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" event={"ID":"a0661629-6e95-40c9-a8d1-4646ae2634f9","Type":"ContainerStarted","Data":"b00ac30a17e71e5f5c4cb5e59ca17acdc9aaf8595949fce8dfbbdc44ef20d2d9"} Oct 31 03:07:04.027288 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:04.027237 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podStartSLOduration=2.027224074 podStartE2EDuration="2.027224074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:04.024841114 +0000 UTC m=+705.705367243" watchObservedRunningTime="2026-10-31 03:07:04.027224074 +0000 UTC m=+705.707750200" Oct 31 03:07:04.772940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:04.772907 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:11.967489 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:11.967455 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8"] Oct 31 03:07:11.971459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:11.971431 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:11.983505 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:11.981207 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"openshift-service-ca.crt\"" Oct 31 03:07:11.983505 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:11.981287 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8"] Oct 31 03:07:11.989553 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:11.986667 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"default-dockercfg-cwqcr\"" Oct 31 03:07:11.989553 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:11.980285 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"kube-root-ca.crt\"" Oct 31 03:07:11.989553 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:11.987456 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"premium-w2-85cb59-kserve-self-signed-certs\"" Oct 31 03:07:12.078008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.077960 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8cf\" (UniqueName: \"kubernetes.io/projected/142a5e7e-e192-401a-9a93-a97d77e11f27-kube-api-access-tl8cf\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.078214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.078031 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-model-cache\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.078214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.078171 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-dshm\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.078320 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.078259 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-tmp-dir\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.078411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.078390 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142a5e7e-e192-401a-9a93-a97d77e11f27-tls-certs\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.078483 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.078465 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-home\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.179213 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.179170 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8cf\" (UniqueName: \"kubernetes.io/projected/142a5e7e-e192-401a-9a93-a97d77e11f27-kube-api-access-tl8cf\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.179423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.179234 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-model-cache\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.179423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.179265 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-dshm\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.179423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.179306 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-tmp-dir\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.179423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.179373 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142a5e7e-e192-401a-9a93-a97d77e11f27-tls-certs\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.179626 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.179428 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-home\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.179767 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.179740 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-model-cache\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.179836 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.179802 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-home\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.180031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.180013 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-tmp-dir\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.182428 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.182399 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142a5e7e-e192-401a-9a93-a97d77e11f27-tls-certs\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.182582 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.182561 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-dshm\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.193190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.193159 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8cf\" (UniqueName: \"kubernetes.io/projected/142a5e7e-e192-401a-9a93-a97d77e11f27-kube-api-access-tl8cf\") pod \"premium-w2-85cb59-kserve-5bbcc7775b-bz4x8\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.291125 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.291019 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:12.570302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.570271 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8"] Oct 31 03:07:12.571271 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:07:12.571247 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod142a5e7e_e192_401a_9a93_a97d77e11f27.slice/crio-b5aaa1dbd0b298500048977fbaf7c79edc740b86243e387ac4ece3cdd412ae8f WatchSource:0}: Error finding container b5aaa1dbd0b298500048977fbaf7c79edc740b86243e387ac4ece3cdd412ae8f: Status 404 returned error can't find the container with id b5aaa1dbd0b298500048977fbaf7c79edc740b86243e387ac4ece3cdd412ae8f Oct 31 03:07:12.772262 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.772224 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:12.784961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:12.784934 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:13.063747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:13.063709 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" event={"ID":"142a5e7e-e192-401a-9a93-a97d77e11f27","Type":"ContainerStarted","Data":"c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47"} Oct 31 03:07:13.064267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:13.063762 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" event={"ID":"142a5e7e-e192-401a-9a93-a97d77e11f27","Type":"ContainerStarted","Data":"b5aaa1dbd0b298500048977fbaf7c79edc740b86243e387ac4ece3cdd412ae8f"} Oct 31 03:07:13.076219 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:13.076188 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:07:13.089854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:13.089797 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podStartSLOduration=2.089779051 podStartE2EDuration="2.089779051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:13.08565555 +0000 UTC m=+714.766181679" watchObservedRunningTime="2026-10-31 03:07:13.089779051 +0000 UTC m=+714.770305178" Oct 31 03:07:13.291171 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:13.291135 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:22.292240 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:22.292197 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:22.305528 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:22.305497 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:23.002468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.002434 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp"] Oct 31 03:07:23.009833 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.008247 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.013190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.013164 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"kube-root-ca.crt\"" Oct 31 03:07:23.013674 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.013650 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp"] Oct 31 03:07:23.013884 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.013866 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"default-dockercfg-76j64\"" Oct 31 03:07:23.015286 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.015254 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"openshift-service-ca.crt\"" Oct 31 03:07:23.015393 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.015326 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"premium-w0-edb9e6-kserve-self-signed-certs\"" Oct 31 03:07:23.100541 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.100507 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-dshm\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.100541 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.100549 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-home\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.100818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.100574 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82wbb\" (UniqueName: \"kubernetes.io/projected/8fb148a9-9b6c-42ab-a831-dd992755ad73-kube-api-access-82wbb\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.100818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.100729 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-model-cache\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.100818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.100781 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-tmp-dir\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.100983 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.100903 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb148a9-9b6c-42ab-a831-dd992755ad73-tls-certs\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.157279 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.157242 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:07:23.201571 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.201529 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-model-cache\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.201768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.201641 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-tmp-dir\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.201768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.201716 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb148a9-9b6c-42ab-a831-dd992755ad73-tls-certs\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.201911 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.201777 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-dshm\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.201911 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.201826 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-home\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.201911 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.201851 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82wbb\" (UniqueName: \"kubernetes.io/projected/8fb148a9-9b6c-42ab-a831-dd992755ad73-kube-api-access-82wbb\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.202175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.201940 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-model-cache\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.202600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.202427 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-home\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.202772 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.202747 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-tmp-dir\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.204555 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.204408 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb148a9-9b6c-42ab-a831-dd992755ad73-tls-certs\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.207937 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.207908 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-dshm\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.233385 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.233358 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82wbb\" (UniqueName: \"kubernetes.io/projected/8fb148a9-9b6c-42ab-a831-dd992755ad73-kube-api-access-82wbb\") pod \"premium-w0-edb9e6-kserve-557cf99995-5bpsp\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.325328 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.325242 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:23.611339 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:23.611309 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp"] Oct 31 03:07:23.611991 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:07:23.611965 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fb148a9_9b6c_42ab_a831_dd992755ad73.slice/crio-9d587d5a711f019b8be3dc95f5fc5485dfbdb56808727d3b60470147dd5b99fc WatchSource:0}: Error finding container 9d587d5a711f019b8be3dc95f5fc5485dfbdb56808727d3b60470147dd5b99fc: Status 404 returned error can't find the container with id 9d587d5a711f019b8be3dc95f5fc5485dfbdb56808727d3b60470147dd5b99fc Oct 31 03:07:24.149353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:24.149313 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" event={"ID":"8fb148a9-9b6c-42ab-a831-dd992755ad73","Type":"ContainerStarted","Data":"66f73d4c54771be7b70924060ef637dce1a3996cb24c001fc31f0f974360fa33"} Oct 31 03:07:24.149518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:24.149364 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" event={"ID":"8fb148a9-9b6c-42ab-a831-dd992755ad73","Type":"ContainerStarted","Data":"9d587d5a711f019b8be3dc95f5fc5485dfbdb56808727d3b60470147dd5b99fc"} Oct 31 03:07:24.172169 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:24.172103 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podStartSLOduration=2.17208357 podStartE2EDuration="2.17208357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:24.170880344 +0000 UTC m=+725.851406472" watchObservedRunningTime="2026-10-31 03:07:24.17208357 +0000 UTC m=+725.852609697" Oct 31 03:07:24.325539 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:24.325509 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:32.134763 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:07:32.134717 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:07:33.326090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:33.326054 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:33.339721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:33.339693 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:07:34.241516 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:07:34.241485 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:08:02.133287 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:08:02.133205 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:08:32.134153 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:08:32.134111 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:08:34.605228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.605189 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr"] Oct 31 03:08:34.609319 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.609296 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:34.615095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.615069 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:08:34.615251 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.615185 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:08:34.625021 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.624994 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr"] Oct 31 03:08:34.640112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.640079 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-522c9223-sjrmr\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:34.640257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.640227 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzxt2\" (UniqueName: \"kubernetes.io/projected/92126027-f6de-44d0-bdc1-2c156ecb60de-kube-api-access-nzxt2\") pod \"maas-api-revoke-sub-522c9223-sjrmr\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:34.640318 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.640303 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-522c9223-sjrmr\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:34.741431 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.741396 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-522c9223-sjrmr\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:34.741609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.741487 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzxt2\" (UniqueName: \"kubernetes.io/projected/92126027-f6de-44d0-bdc1-2c156ecb60de-kube-api-access-nzxt2\") pod \"maas-api-revoke-sub-522c9223-sjrmr\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:34.741609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.741525 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-522c9223-sjrmr\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:34.742163 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.742139 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-522c9223-sjrmr\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:34.743826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.743801 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-522c9223-sjrmr\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:34.756090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.756054 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzxt2\" (UniqueName: \"kubernetes.io/projected/92126027-f6de-44d0-bdc1-2c156ecb60de-kube-api-access-nzxt2\") pod \"maas-api-revoke-sub-522c9223-sjrmr\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:34.925781 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:34.925742 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:35.207479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:35.207449 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr"] Oct 31 03:08:35.207611 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:08:35.207508 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92126027_f6de_44d0_bdc1_2c156ecb60de.slice/crio-26b0b720a4781aacfdfc5793ae7f92b83af4159e2d5c57809be73d7f0e28a790 WatchSource:0}: Error finding container 26b0b720a4781aacfdfc5793ae7f92b83af4159e2d5c57809be73d7f0e28a790: Status 404 returned error can't find the container with id 26b0b720a4781aacfdfc5793ae7f92b83af4159e2d5c57809be73d7f0e28a790 Oct 31 03:08:35.618596 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:35.618513 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" event={"ID":"92126027-f6de-44d0-bdc1-2c156ecb60de","Type":"ContainerStarted","Data":"26b0b720a4781aacfdfc5793ae7f92b83af4159e2d5c57809be73d7f0e28a790"} Oct 31 03:08:38.639890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:38.639802 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" event={"ID":"92126027-f6de-44d0-bdc1-2c156ecb60de","Type":"ContainerStarted","Data":"1074d8e1d1f7f22ddd2df32f2b5deb257fd2a1189589e5387cf1f4e391b431b9"} Oct 31 03:08:38.661571 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:38.661516 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" podStartSLOduration=1.804346155 podStartE2EDuration="4.661500506s" totalImagesPullingTime="2.857154351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:38.65989813 +0000 UTC m=+800.340424257" watchObservedRunningTime="2026-10-31 03:08:38.661500506 +0000 UTC m=+800.342026636" Oct 31 03:08:39.649997 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:39.649960 4456 generic.go:356] "Generic (PLEG): container finished" podID="92126027-f6de-44d0-bdc1-2c156ecb60de" containerID="1074d8e1d1f7f22ddd2df32f2b5deb257fd2a1189589e5387cf1f4e391b431b9" exitCode=0 Oct 31 03:08:39.650512 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:39.650080 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" event={"ID":"92126027-f6de-44d0-bdc1-2c156ecb60de","Type":"ContainerDied","Data":"1074d8e1d1f7f22ddd2df32f2b5deb257fd2a1189589e5387cf1f4e391b431b9"} Oct 31 03:08:40.458453 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.458419 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb"] Oct 31 03:08:40.463498 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.463467 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.473126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.473100 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb"] Oct 31 03:08:40.473525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.473501 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"distinct-w5-56afbd-kserve-self-signed-certs\"" Oct 31 03:08:40.476484 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.476463 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"openshift-service-ca.crt\"" Oct 31 03:08:40.476599 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.476550 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"default-dockercfg-gdkj4\"" Oct 31 03:08:40.476671 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.476638 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"kube-root-ca.crt\"" Oct 31 03:08:40.602726 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.602687 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tmp-dir\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.602726 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.602730 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q76qm\" (UniqueName: \"kubernetes.io/projected/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-kube-api-access-q76qm\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.603002 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.602848 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-dshm\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.603002 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.602876 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-model-cache\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.603002 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.602923 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-home\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.603212 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.603113 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tls-certs\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.704417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.704381 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-model-cache\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.704823 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.704470 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-home\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.704823 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.704530 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tls-certs\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.704823 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.704597 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tmp-dir\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.704823 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.704627 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76qm\" (UniqueName: \"kubernetes.io/projected/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-kube-api-access-q76qm\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.704823 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.704713 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-dshm\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.705146 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.704855 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-model-cache\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.705146 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.704905 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-home\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.705320 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.705201 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tmp-dir\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.711795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.711723 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-dshm\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.712120 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.712096 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tls-certs\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.720240 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.720194 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q76qm\" (UniqueName: \"kubernetes.io/projected/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-kube-api-access-q76qm\") pod \"distinct-w5-56afbd-kserve-85dd9b46cd-clnbb\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.789878 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.789843 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:40.834146 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.834116 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:40.906964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.906934 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92126027-f6de-44d0-bdc1-2c156ecb60de-kube-api-access-nzxt2\" (UniqueName: \"kubernetes.io/projected/92126027-f6de-44d0-bdc1-2c156ecb60de-kube-api-access-nzxt2\") pod \"92126027-f6de-44d0-bdc1-2c156ecb60de\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " Oct 31 03:08:40.907166 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.907060 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-openshift-ca\") pod \"92126027-f6de-44d0-bdc1-2c156ecb60de\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " Oct 31 03:08:40.907166 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.907153 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-xks-ca\") pod \"92126027-f6de-44d0-bdc1-2c156ecb60de\" (UID: \"92126027-f6de-44d0-bdc1-2c156ecb60de\") " Oct 31 03:08:40.907937 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.907827 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-openshift-ca" pod "92126027-f6de-44d0-bdc1-2c156ecb60de" (UID: "92126027-f6de-44d0-bdc1-2c156ecb60de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:40.909524 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.909495 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-xks-ca" pod "92126027-f6de-44d0-bdc1-2c156ecb60de" (UID: "92126027-f6de-44d0-bdc1-2c156ecb60de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:40.909869 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:40.909829 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92126027-f6de-44d0-bdc1-2c156ecb60de-kube-api-access-nzxt2" pod "92126027-f6de-44d0-bdc1-2c156ecb60de" (UID: "92126027-f6de-44d0-bdc1-2c156ecb60de"). InnerVolumeSpecName "kube-api-access-nzxt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:41.008929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.008838 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:41.008929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.008870 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzxt2\" (UniqueName: \"kubernetes.io/projected/92126027-f6de-44d0-bdc1-2c156ecb60de-kube-api-access-nzxt2\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:41.008929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.008881 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92126027-f6de-44d0-bdc1-2c156ecb60de-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:41.082315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.082283 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb"] Oct 31 03:08:41.083926 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:08:41.083893 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea1a1ec2_20e7_4119_a3a8_82a8d6142d51.slice/crio-1c92b51be7cdd32363d96b008e922e50fd65b002935507d51bafdc69ec66b57b WatchSource:0}: Error finding container 1c92b51be7cdd32363d96b008e922e50fd65b002935507d51bafdc69ec66b57b: Status 404 returned error can't find the container with id 1c92b51be7cdd32363d96b008e922e50fd65b002935507d51bafdc69ec66b57b Oct 31 03:08:41.665004 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.664970 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" Oct 31 03:08:41.665193 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.664970 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr" event={"ID":"92126027-f6de-44d0-bdc1-2c156ecb60de","Type":"ContainerDied","Data":"26b0b720a4781aacfdfc5793ae7f92b83af4159e2d5c57809be73d7f0e28a790"} Oct 31 03:08:41.665193 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.665117 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26b0b720a4781aacfdfc5793ae7f92b83af4159e2d5c57809be73d7f0e28a790" Oct 31 03:08:41.666970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.666946 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" event={"ID":"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51","Type":"ContainerStarted","Data":"d28822ff6f0e189d0f41ea0ff8acd7748ae3c4c29bd5941e088e236c963081cb"} Oct 31 03:08:41.667122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.666982 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" event={"ID":"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51","Type":"ContainerStarted","Data":"1c92b51be7cdd32363d96b008e922e50fd65b002935507d51bafdc69ec66b57b"} Oct 31 03:08:41.689991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.689939 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podStartSLOduration=1.689928043 podStartE2EDuration="1.689928043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:41.689412368 +0000 UTC m=+803.369938492" watchObservedRunningTime="2026-10-31 03:08:41.689928043 +0000 UTC m=+803.370454169" Oct 31 03:08:41.791192 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:41.791071 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:42.667327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.667287 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84"] Oct 31 03:08:42.667825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.667806 4456 cpu_manager.go:414] "containerMap: removing container" podUID="92126027-f6de-44d0-bdc1-2c156ecb60de" containerName="revoke-keys" Oct 31 03:08:42.667825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.667826 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92126027-f6de-44d0-bdc1-2c156ecb60de" containerName="revoke-keys" Oct 31 03:08:42.667951 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.667938 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="92126027-f6de-44d0-bdc1-2c156ecb60de" containerName="revoke-keys" Oct 31 03:08:42.671555 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.671531 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.679977 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.679955 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bebea4-openshift-default-dockercfg-wg8mv\"" Oct 31 03:08:42.687394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.687368 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84"] Oct 31 03:08:42.724890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.724852 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.724890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.724891 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.725135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.724920 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.725135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.725065 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.725214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.725161 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.725214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.725188 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.725301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.725287 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.725357 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.725345 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.725389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.725368 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzcw\" (UniqueName: \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-kube-api-access-hbzcw\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.826932 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.826880 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.826932 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.826937 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzcw\" (UniqueName: \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-kube-api-access-hbzcw\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.826993 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827010 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827057 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827079 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827121 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827162 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827211 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827582 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827605 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827687 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827706 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.827957 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.827935 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.829496 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.829473 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.829606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.829506 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.848301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.848276 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.850401 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.850376 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzcw\" (UniqueName: \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-kube-api-access-hbzcw\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.901399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.901357 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp"] Oct 31 03:08:42.906529 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.906505 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:42.925015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.924922 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-924ee5-openshift-default-dockercfg-w4fmh\"" Oct 31 03:08:42.931495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.931468 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp"] Oct 31 03:08:42.987702 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.987667 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:42.999273 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:42.999249 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g"] Oct 31 03:08:43.003145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.003129 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.018286 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.018055 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g"] Oct 31 03:08:43.029168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.028457 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-99bc97-openshift-default-dockercfg-pzvq9\"" Oct 31 03:08:43.029168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.028711 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvnp\" (UniqueName: \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-kube-api-access-6wvnp\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.029168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.028740 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.029168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.028767 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.029168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.028789 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.029168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.028832 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.029168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.028863 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.029168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.028888 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.029168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.028907 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.029168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.028932 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.130317 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130277 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.130481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130327 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.130481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130437 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvnp\" (UniqueName: \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-kube-api-access-6wvnp\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.130481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130471 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.130636 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130499 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.130636 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130529 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.130636 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130564 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4657dd4e-e0de-4400-8304-74c7794421f5-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.130636 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130596 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.130826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130644 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.130826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130701 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.130826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130728 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.130826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130753 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.130826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130770 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4657dd4e-e0de-4400-8304-74c7794421f5-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.130826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130789 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.130826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130810 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.130826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130828 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.131278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130898 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.131278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.130921 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb4ch\" (UniqueName: \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-kube-api-access-tb4ch\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.131278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.131086 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.131278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.131105 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.131278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.131262 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.131518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.131364 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.131518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.131400 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.133167 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.133144 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.133380 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.133359 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.143278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.143250 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.143386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.143371 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvnp\" (UniqueName: \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-kube-api-access-6wvnp\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.223608 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.223514 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:43.232463 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.232426 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.232644 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.232477 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.232644 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.232563 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4657dd4e-e0de-4400-8304-74c7794421f5-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.232644 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.232620 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.232859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.232694 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.232859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.232732 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4657dd4e-e0de-4400-8304-74c7794421f5-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.232859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.232760 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.233001 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.232857 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.233001 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.232891 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb4ch\" (UniqueName: \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-kube-api-access-tb4ch\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.233394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.233371 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.234016 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.233990 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4657dd4e-e0de-4400-8304-74c7794421f5-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.234262 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.234243 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.234474 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.234456 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.234689 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.234672 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.235795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.235768 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4657dd4e-e0de-4400-8304-74c7794421f5-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.236193 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.236175 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.245844 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.245821 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.246049 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.246021 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb4ch\" (UniqueName: \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-kube-api-access-tb4ch\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.267511 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.267476 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x"] Oct 31 03:08:43.273429 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.273402 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.311177 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:08:43.311147 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda341e75_5d33_4a52_b018_bf1b2d2f83e9.slice/crio-da275a0c66812cbc60d3238ecef58648b0949f36d3d59e244a30b070bb9fe8b5 WatchSource:0}: Error finding container da275a0c66812cbc60d3238ecef58648b0949f36d3d59e244a30b070bb9fe8b5: Status 404 returned error can't find the container with id da275a0c66812cbc60d3238ecef58648b0949f36d3d59e244a30b070bb9fe8b5 Oct 31 03:08:43.312366 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.312342 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7d8daf-openshift-default-dockercfg-lm6g8\"" Oct 31 03:08:43.314197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.314161 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.314436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.314353 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.314436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.314407 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.315152 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.314435 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x"] Oct 31 03:08:43.319108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.319084 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84"] Oct 31 03:08:43.335186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.334248 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.335186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.334304 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.335186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.334422 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.335186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.334447 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chcw2\" (UniqueName: \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-kube-api-access-chcw2\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.335186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.334474 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.335186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.334509 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.335186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.334533 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.335186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.334592 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133bc9f0-a624-473c-84a1-c2db415a52aa-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.335186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.334642 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.335186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.334975 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:43.435584 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.435544 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.435739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.435600 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.435739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.435670 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.435739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.435695 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chcw2\" (UniqueName: \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-kube-api-access-chcw2\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.435739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.435724 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.435987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.435759 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.435987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.435783 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.435987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.435840 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133bc9f0-a624-473c-84a1-c2db415a52aa-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.435987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.435889 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.436253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.436231 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.436291 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.436262 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.437689 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.436805 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.437689 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.437222 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133bc9f0-a624-473c-84a1-c2db415a52aa-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.437972 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.437931 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.439393 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.439352 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.440660 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.440631 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.458720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.458682 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chcw2\" (UniqueName: \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-kube-api-access-chcw2\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.459142 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.459118 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.524707 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.524678 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp"] Oct 31 03:08:43.525583 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:08:43.525553 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371d1f98_d538_49f5_9e4e_04cbb3739b4b.slice/crio-f2b94841a43486fb2a15ce2ae86ff6fab2e21c3022e643331a2ebee40e04813f WatchSource:0}: Error finding container f2b94841a43486fb2a15ce2ae86ff6fab2e21c3022e643331a2ebee40e04813f: Status 404 returned error can't find the container with id f2b94841a43486fb2a15ce2ae86ff6fab2e21c3022e643331a2ebee40e04813f Oct 31 03:08:43.528114 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.528081 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.528197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.528146 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.528197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.528177 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.601709 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.601679 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:43.655730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.654736 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g"] Oct 31 03:08:43.658259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.656724 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.658259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.656796 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.658259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.656838 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.683641 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.683599 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" event={"ID":"4657dd4e-e0de-4400-8304-74c7794421f5","Type":"ContainerStarted","Data":"cb1a972a226b129b3593b2f426260572aa8ce61e9e63d1ca51290712770218a5"} Oct 31 03:08:43.694247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.686808 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" event={"ID":"da341e75-5d33-4a52-b018-bf1b2d2f83e9","Type":"ContainerStarted","Data":"c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869"} Oct 31 03:08:43.694247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.686873 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" event={"ID":"da341e75-5d33-4a52-b018-bf1b2d2f83e9","Type":"ContainerStarted","Data":"da275a0c66812cbc60d3238ecef58648b0949f36d3d59e244a30b070bb9fe8b5"} Oct 31 03:08:43.728385 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.720156 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" event={"ID":"371d1f98-d538-49f5-9e4e-04cbb3739b4b","Type":"ContainerStarted","Data":"37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa"} Oct 31 03:08:43.728385 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.720214 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" event={"ID":"371d1f98-d538-49f5-9e4e-04cbb3739b4b","Type":"ContainerStarted","Data":"f2b94841a43486fb2a15ce2ae86ff6fab2e21c3022e643331a2ebee40e04813f"} Oct 31 03:08:43.731233 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.731174 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podStartSLOduration=1.731156954 podStartE2EDuration="1.731156954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:43.722444266 +0000 UTC m=+805.402970401" watchObservedRunningTime="2026-10-31 03:08:43.731156954 +0000 UTC m=+805.411683082" Oct 31 03:08:43.747713 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.747654 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podStartSLOduration=1.747636155 podStartE2EDuration="1.747636155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:43.747194729 +0000 UTC m=+805.427720867" watchObservedRunningTime="2026-10-31 03:08:43.747636155 +0000 UTC m=+805.428162283" Oct 31 03:08:43.935465 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.935435 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x"] Oct 31 03:08:43.937847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.937810 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.937948 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.937891 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.937948 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.937937 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:08:43.988005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:43.987978 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:44.225054 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:44.225013 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:44.228431 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:44.228400 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:44.228604 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:44.228586 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:44.740277 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:44.740229 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" event={"ID":"133bc9f0-a624-473c-84a1-c2db415a52aa","Type":"ContainerStarted","Data":"4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a"} Oct 31 03:08:44.740574 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:44.740550 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" event={"ID":"133bc9f0-a624-473c-84a1-c2db415a52aa","Type":"ContainerStarted","Data":"9937659e80cad8671e2c0e4958746175df0309bf8a372d7e0257743255ae9033"} Oct 31 03:08:44.771568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:44.768400 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podStartSLOduration=1.768379532 podStartE2EDuration="1.768379532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:44.765539931 +0000 UTC m=+806.446066060" watchObservedRunningTime="2026-10-31 03:08:44.768379532 +0000 UTC m=+806.448905660" Oct 31 03:08:44.787060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:44.784703 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" event={"ID":"4657dd4e-e0de-4400-8304-74c7794421f5","Type":"ContainerStarted","Data":"7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9"} Oct 31 03:08:45.044138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.034176 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:08:45.044138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.034239 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:08:45.338226 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.338086 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:45.417077 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.393304 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:45.417077 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.393371 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:45.443063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.418213 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:08:45.443063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.418363 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:08:45.612178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.612085 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:45.638206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.638171 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:45.638492 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.638464 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:46.016062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.999435 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:08:46.016062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:45.999519 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:08:46.229063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:46.228247 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:46.229063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:46.228303 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:46.341121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:46.336649 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:08:46.341121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:46.336715 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:08:46.606655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:46.606141 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:46.606655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:46.606212 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:47.001062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.000187 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:08:47.001062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.000246 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:08:47.081075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.079056 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:47.147080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.146631 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:47.228542 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.227397 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:47.228542 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.228418 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:47.228542 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.228455 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:47.341111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.337417 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:08:47.341111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.337479 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:08:47.607140 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.606568 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:47.607140 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.606628 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:47.990475 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.990437 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:08:47.990738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:47.990711 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:08:48.230059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:48.225474 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:48.230059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:48.225542 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:48.230338 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:48.230122 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:48.234066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:48.230602 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:08:48.268110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:48.267494 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podStartSLOduration=6.267473851 podStartE2EDuration="6.267473851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:44.830380707 +0000 UTC m=+806.510906847" watchObservedRunningTime="2026-10-31 03:08:48.267473851 +0000 UTC m=+809.947999979" Oct 31 03:08:48.355060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:48.352681 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:08:48.355060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:48.352742 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:08:48.620118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:48.606133 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:48.620118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:48.606251 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:49.005067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.004969 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:08:49.005067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.005028 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:08:49.156189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.156137 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 03:08:49.225061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.224489 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:49.225061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.224556 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:49.233060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.230423 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:49.344661 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.344562 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:08:49.345614 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.344993 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:08:49.748061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.730538 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:49.799063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.775361 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:49.998143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.988550 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:08:49.998143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:49.988613 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:08:50.070058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.069114 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:50.164179 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.164128 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:50.169060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.165131 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:08:50.245202 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.245162 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:50.254160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.245461 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:50.284073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.279190 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:50.338565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.338529 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:08:50.347096 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.339013 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:08:50.602112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.601237 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7r8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:08:50.602112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.601315 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" podUID="ef33ec8e-50ff-4f0b-a84c-25a9b07c2221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:50.603117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.603052 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:50.603316 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.603292 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:50.802247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.802217 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:50.822489 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.822457 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:50.995110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.995064 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:08:50.995358 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:50.995327 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:08:51.086211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.086166 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:51.087291 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.087270 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:08:51.109353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.109329 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:08:51.193064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.192926 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:51.234661 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.234138 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:51.234661 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.234212 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:51.295513 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.288125 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:08:51.339138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.339105 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:08:51.339398 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.339371 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:08:51.423426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.423393 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:08:51.513697 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.513666 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:08:51.611700 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.611606 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:51.617063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.611946 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:51.782803 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.782745 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:51.995063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.989370 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:08:51.995063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:51.989429 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:08:52.054064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.053300 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:08:52.239065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.226711 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:52.243168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.243127 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:52.336173 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.335640 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:08:52.336173 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.335700 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:08:52.515208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.515133 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:52.616119 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.611285 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:52.616119 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.611439 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:52.989238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.989174 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:52.991066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.990940 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:08:52.991278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:52.991256 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:08:53.229443 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.229403 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:53.234060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.229990 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:53.289204 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.285359 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:53.353061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.335609 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:08:53.370576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.370533 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:08:53.536298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.536237 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:53.536985 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.536965 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:08:53.602526 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.602436 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:53.602768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.602740 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:53.784133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.784088 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:53.784700 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:53.784682 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:08:54.084064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.082701 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:08:54.181061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.176995 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:54.227062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.224399 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:08:54.227062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.224460 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:08:54.288060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.287552 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:54.288553 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.288210 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:08:54.330062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.327747 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:54.337641 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.337551 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:08:54.337917 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.337875 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:08:54.595629 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.595546 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:08:54.610456 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.610422 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:54.610717 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.610695 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:54.997600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.997563 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:08:54.997904 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:54.997876 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:55.064874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.064843 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb"] Oct 31 03:08:55.131112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.131081 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb"] Oct 31 03:08:55.201121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.136175 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.268590 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.265757 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"distinct-w6-7d8daf-kserve-self-signed-certs\"" Oct 31 03:08:55.306234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.306199 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-model-cache\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.308760 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.308731 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-dshm\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.309006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.308985 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-home\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.309202 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.309184 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtb9\" (UniqueName: \"kubernetes.io/projected/ab9ee55b-39a4-48da-a929-20cbb3f41730-kube-api-access-5xtb9\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.309356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.309341 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ee55b-39a4-48da-a929-20cbb3f41730-tls-certs\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.309686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.309669 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-tmp-dir\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.410616 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.410577 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-model-cache\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.411458 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.411432 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-dshm\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.411981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.411960 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-home\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.412176 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.412152 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtb9\" (UniqueName: \"kubernetes.io/projected/ab9ee55b-39a4-48da-a929-20cbb3f41730-kube-api-access-5xtb9\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.412653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.412634 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ee55b-39a4-48da-a929-20cbb3f41730-tls-certs\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.422318 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.422296 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-dshm\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.422465 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.411192 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-model-cache\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.426090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.412398 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-home\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.426677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.426657 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-tmp-dir\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.427454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.427028 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-tmp-dir\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.436657 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.436634 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ee55b-39a4-48da-a929-20cbb3f41730-tls-certs\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.437449 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.437398 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtb9\" (UniqueName: \"kubernetes.io/projected/ab9ee55b-39a4-48da-a929-20cbb3f41730-kube-api-access-5xtb9\") pod \"distinct-w6-7d8daf-kserve-7bcd9754f9-njknb\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:55.605545 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.605442 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:08:55.605831 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.605784 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:08:55.610292 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:55.610261 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:56.049084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.046781 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:08:56.049084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.046857 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:56.050796 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.050767 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:56.052335 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.052313 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:56.066531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.066497 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:08:56.182926 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:08:56.182314 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab9ee55b_39a4_48da_a929_20cbb3f41730.slice/crio-01d1a4a7a9673f1a15df2ef4f1b9b98c0a69f981fa3a0ef5c660ed7d06b1999d WatchSource:0}: Error finding container 01d1a4a7a9673f1a15df2ef4f1b9b98c0a69f981fa3a0ef5c660ed7d06b1999d: Status 404 returned error can't find the container with id 01d1a4a7a9673f1a15df2ef4f1b9b98c0a69f981fa3a0ef5c660ed7d06b1999d Oct 31 03:08:56.191200 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.190978 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb"] Oct 31 03:08:56.267607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.267264 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:08:56.267607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.267327 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:56.299094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.295873 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" event={"ID":"ab9ee55b-39a4-48da-a929-20cbb3f41730","Type":"ContainerStarted","Data":"a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431"} Oct 31 03:08:56.299094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.295932 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" event={"ID":"ab9ee55b-39a4-48da-a929-20cbb3f41730","Type":"ContainerStarted","Data":"01d1a4a7a9673f1a15df2ef4f1b9b98c0a69f981fa3a0ef5c660ed7d06b1999d"} Oct 31 03:08:56.385571 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.376757 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podStartSLOduration=1.376735966 podStartE2EDuration="1.376735966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:56.356952138 +0000 UTC m=+818.037478269" watchObservedRunningTime="2026-10-31 03:08:56.376735966 +0000 UTC m=+818.057262094" Oct 31 03:08:56.385571 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.385158 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:08:56.385571 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.385253 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:56.614122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.613645 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:08:56.855096 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.853365 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:56.855096 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.854757 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:56.856863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.856839 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:08:56.887154 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.883364 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:56.887154 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.884524 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:56.891287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:56.891258 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:08:57.195356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:57.195314 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:57.196417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:57.196393 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:08:57.197171 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:08:57.197149 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:09:02.135225 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:09:02.135183 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:09:02.141505 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.141472 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp"] Oct 31 03:09:02.146418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.146390 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.184861 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.184826 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp"] Oct 31 03:09:02.246609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.246565 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.246828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.246640 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.246828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.246713 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.246828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.246794 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.246991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.246829 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.247069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.246984 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.247069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.247024 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-278sx\" (UniqueName: \"kubernetes.io/projected/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-kube-api-access-278sx\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.247180 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.247133 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.247238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.247188 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.348734 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.348697 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.348930 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.348745 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-278sx\" (UniqueName: \"kubernetes.io/projected/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-kube-api-access-278sx\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.348930 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.348792 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.348930 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.348827 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.348930 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.348852 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.348930 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.348882 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.348930 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.348899 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.348930 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.348926 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.349332 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.348959 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.349389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.349326 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.349858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.349585 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.349858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.349693 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.350195 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.350172 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.350309 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.350287 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.352021 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.351996 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.352564 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.352542 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.367333 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.367066 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.369015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.368224 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-278sx\" (UniqueName: \"kubernetes.io/projected/29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3-kube-api-access-278sx\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6sbp\" (UID: \"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.474627 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.474581 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:02.774677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.774628 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:02.774792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.774725 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:02.774792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.774766 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:02.784567 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:02.784536 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp"] Oct 31 03:09:03.354064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:03.347221 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" event={"ID":"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3","Type":"ContainerStarted","Data":"bf91dfe7576d89a886858e129eba6c171f95a13673b0853454221d70713d98ef"} Oct 31 03:09:03.354064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:03.347282 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" event={"ID":"29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3","Type":"ContainerStarted","Data":"c90ec8426ae27690fcbc7f84e2dacc9420bd87cc5f17e874137e4f25b4d38795"} Oct 31 03:09:03.394215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:03.394151 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" podStartSLOduration=1.394128688 podStartE2EDuration="1.394128688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:03.376551595 +0000 UTC m=+825.057077734" watchObservedRunningTime="2026-10-31 03:09:03.394128688 +0000 UTC m=+825.074654870" Oct 31 03:09:03.477062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:03.476412 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:03.484314 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:03.484249 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6sbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:03.484513 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:03.484337 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" podUID="29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:09:04.476062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:04.475081 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6sbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:04.476062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:04.475152 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" podUID="29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:09:05.478540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:05.478500 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6sbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:05.479287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:05.479250 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" podUID="29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:09:05.623062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:05.622538 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:09:05.657193 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:05.657159 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:09:06.440993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:06.440967 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:09:06.477595 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:06.477549 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6sbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:06.477891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:06.477862 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" podUID="29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:09:07.475162 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:07.475124 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6sbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:07.475761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:07.475720 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" podUID="29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:09:08.475501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:08.475461 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6sbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:08.476136 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:08.476102 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" podUID="29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:09:09.475099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:09.475060 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6sbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:09.475299 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:09.475125 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" podUID="29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:09:10.475859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:10.475823 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6sbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:10.476396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:10.475885 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" podUID="29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:09:11.061287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.061228 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm"] Oct 31 03:09:11.077909 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.077881 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm"] Oct 31 03:09:11.078252 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.078185 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.090253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.090159 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"distinct-2-w5-56afbd-kserve-self-signed-certs\"" Oct 31 03:09:11.197383 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.197345 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6x95\" (UniqueName: \"kubernetes.io/projected/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-kube-api-access-m6x95\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.197649 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.197625 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tls-certs\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.197793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.197778 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-home\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.197939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.197925 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-model-cache\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.198098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.198079 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tmp-dir\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.201064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.198309 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-dshm\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.304863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.304820 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tmp-dir\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.306678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.306648 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-dshm\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.307328 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.307308 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6x95\" (UniqueName: \"kubernetes.io/projected/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-kube-api-access-m6x95\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.307812 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.305517 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tmp-dir\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.308005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.307985 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tls-certs\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.308656 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.308637 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-home\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.309204 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.309074 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-home\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.309390 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.309371 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-model-cache\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.309886 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.309782 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-model-cache\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.316214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.315838 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tls-certs\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.316640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.316595 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-dshm\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.322273 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.322227 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6x95\" (UniqueName: \"kubernetes.io/projected/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-kube-api-access-m6x95\") pod \"distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.441086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.441051 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:11.688028 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.687990 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:11.688859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.688835 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:11.690196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.690175 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" Oct 31 03:09:11.758411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:11.758377 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm"] Oct 31 03:09:12.469527 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:12.469486 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" event={"ID":"a0a17bda-c051-44fb-bc97-1b3aa1a4c290","Type":"ContainerStarted","Data":"818fbff4aae0dcceb2238d6a349b2e246c63d28a7ce22bfdec443d4e5c299d2f"} Oct 31 03:09:12.469731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:12.469537 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" event={"ID":"a0a17bda-c051-44fb-bc97-1b3aa1a4c290","Type":"ContainerStarted","Data":"0820a010e9f12c33ee9ad140cf16b09d6815c4cc1b7762c5f6725dd040a10fb2"} Oct 31 03:09:12.493353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:12.493280 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podStartSLOduration=1.493260501 podStartE2EDuration="1.493260501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:12.491466665 +0000 UTC m=+834.171992811" watchObservedRunningTime="2026-10-31 03:09:12.493260501 +0000 UTC m=+834.173786630" Oct 31 03:09:12.891176 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:12.891099 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7"] Oct 31 03:09:12.896192 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:12.896163 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:12.935617 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:12.935589 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-56afbd-openshift-default-dockercfg-xms2n\"" Oct 31 03:09:12.971871 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:12.971062 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7"] Oct 31 03:09:13.032250 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.032209 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.032436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.032326 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.032436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.032381 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.032436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.032410 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.032605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.032492 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.032605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.032584 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.032711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.032629 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.032770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.032739 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.032822 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.032780 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9t9\" (UniqueName: \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-kube-api-access-5r9t9\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.133701 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.133659 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.133900 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.133725 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.133900 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.133777 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.133900 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.133813 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.133900 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.133863 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.134215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.133918 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.134215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.133954 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.134215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.134006 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.134215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.134073 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9t9\" (UniqueName: \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-kube-api-access-5r9t9\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.134215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.134158 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.134215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.134190 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.134629 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.134575 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.134629 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.134603 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.134629 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.134617 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.136446 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.136418 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.136610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.136586 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.150074 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.149979 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.150388 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.150369 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9t9\" (UniqueName: \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-kube-api-access-5r9t9\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.216643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.216603 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:13.441590 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.441551 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:13.511239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.511209 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7"] Oct 31 03:09:13.513014 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:09:13.512990 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a06c62_e2ef_4e7d_8763_b7bf9111d163.slice/crio-269b35cb1fde1e7ee8ea2baed2023fec21a6a02f6e41ae5a48abda672eb93261 WatchSource:0}: Error finding container 269b35cb1fde1e7ee8ea2baed2023fec21a6a02f6e41ae5a48abda672eb93261: Status 404 returned error can't find the container with id 269b35cb1fde1e7ee8ea2baed2023fec21a6a02f6e41ae5a48abda672eb93261 Oct 31 03:09:13.515921 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.515877 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:13.516019 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.515968 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:13.516101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:13.516015 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:14.488330 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:14.488292 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" event={"ID":"74a06c62-e2ef-4e7d-8763-b7bf9111d163","Type":"ContainerStarted","Data":"f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf"} Oct 31 03:09:14.488795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:14.488360 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" event={"ID":"74a06c62-e2ef-4e7d-8763-b7bf9111d163","Type":"ContainerStarted","Data":"269b35cb1fde1e7ee8ea2baed2023fec21a6a02f6e41ae5a48abda672eb93261"} Oct 31 03:09:14.517726 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:14.517665 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" podStartSLOduration=2.517652992 podStartE2EDuration="2.517652992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:14.516804797 +0000 UTC m=+836.197330923" watchObservedRunningTime="2026-10-31 03:09:14.517652992 +0000 UTC m=+836.198179119" Oct 31 03:09:15.217002 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:15.216962 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:15.218707 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:15.218681 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:15.218804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:15.218754 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:09:16.217083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:16.217020 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:16.217606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:16.217116 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:09:17.217391 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:17.217349 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:17.217919 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:17.217429 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:09:18.217482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:18.217441 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:18.217876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:18.217510 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:09:19.217493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:19.217341 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:19.217493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:19.217407 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:09:20.576579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:20.576504 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:20.577613 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:20.577552 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:20.578689 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:20.578666 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:09:21.441844 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:21.441809 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:21.455090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:21.455059 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:21.564717 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:21.564682 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:09:28.108165 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.108119 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv"] Oct 31 03:09:28.114238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.114215 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.129910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.129883 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv"] Oct 31 03:09:28.206087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.206031 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xp6r\" (UniqueName: \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-kube-api-access-7xp6r\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.206272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.206160 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d087ee5-7714-4cc3-88a8-de6e13297913-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.206272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.206224 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.206423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.206395 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.206482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.206452 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.206559 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.206543 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.206605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.206582 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.206687 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.206672 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.206788 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.206773 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.308533 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.308498 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.308711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.308542 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.308711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.308579 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.308711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.308636 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.308711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.308666 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xp6r\" (UniqueName: \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-kube-api-access-7xp6r\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.309011 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.308717 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d087ee5-7714-4cc3-88a8-de6e13297913-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.309011 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.308766 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.309011 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.308831 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.309011 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.308871 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.309295 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.309082 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.310285 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.310239 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d087ee5-7714-4cc3-88a8-de6e13297913-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.310543 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.310519 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.310949 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.310918 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.317201 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.313636 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.320088 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.319991 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs"] Oct 31 03:09:28.325817 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.325411 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.327662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.327642 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.330995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.330971 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.388102 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.387692 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs"] Oct 31 03:09:28.491491 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.490253 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-edb9e6-openshift-default-dockercfg-n4zc9\"" Oct 31 03:09:28.501243 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.500869 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.501764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.501733 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.501898 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.501795 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.501898 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.501884 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhz8q\" (UniqueName: \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-kube-api-access-zhz8q\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.502005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.501914 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.502005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.501937 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f59a4f5-d136-4c9b-a557-792bb6b71657-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.502005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.501967 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.502181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.502024 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.502181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.502116 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.505587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.505532 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.516064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.508595 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xp6r\" (UniqueName: \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-kube-api-access-7xp6r\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.533981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.533820 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5"] Oct 31 03:09:28.539079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.538680 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5"] Oct 31 03:09:28.539079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.538796 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.603464 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603425 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.603675 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603483 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.603675 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603503 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.603675 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603655 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhz8q\" (UniqueName: \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-kube-api-access-zhz8q\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.603854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603702 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.603854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603741 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.603854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603766 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f59a4f5-d136-4c9b-a557-792bb6b71657-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.603854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603795 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.603854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603852 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.604150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603948 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.604150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.603989 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.604150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604029 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.604150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604079 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.604150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604110 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.604315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604162 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.604315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604168 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.604441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604421 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.604544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604521 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f59a4f5-d136-4c9b-a557-792bb6b71657-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.604856 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604828 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.604856 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604847 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.605080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604934 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.605080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604968 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.605080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.604999 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdh7b\" (UniqueName: \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-kube-api-access-hdh7b\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.606857 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.606586 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.606857 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.606817 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.620948 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.620916 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.622827 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.622793 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhz8q\" (UniqueName: \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-kube-api-access-zhz8q\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.653405 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.653367 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:28.706143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.706101 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.706354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.706154 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdh7b\" (UniqueName: \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-kube-api-access-hdh7b\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.706354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.706214 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.706354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.706244 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.706354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.706302 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.706594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.706427 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.706594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.706468 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.706594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.706501 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.706594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.706537 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.706594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.706551 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.707435 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.707216 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.707598 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.707445 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.707756 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.707731 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.708270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.708245 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.709531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.709504 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.711858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.711832 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.724173 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.723839 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.724325 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.724297 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdh7b\" (UniqueName: \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-kube-api-access-hdh7b\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.769428 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.769171 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:28.875355 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.875317 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:28.961030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.960120 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs"] Oct 31 03:09:28.968410 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:09:28.968349 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f59a4f5_d136_4c9b_a557_792bb6b71657.slice/crio-63d8c35befae4b44ebab085ded9b8db8eb334fddcda2c66aec27df941320e83d WatchSource:0}: Error finding container 63d8c35befae4b44ebab085ded9b8db8eb334fddcda2c66aec27df941320e83d: Status 404 returned error can't find the container with id 63d8c35befae4b44ebab085ded9b8db8eb334fddcda2c66aec27df941320e83d Oct 31 03:09:28.974655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.973257 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:28.974655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.973348 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:28.974655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:28.973390 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:29.075086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.075056 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv"] Oct 31 03:09:29.080747 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:09:29.080721 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d087ee5_7714_4cc3_88a8_de6e13297913.slice/crio-d7b2d290acd75f87632518d71155e48c58ea0cec4d1f9b270bc9bb1601f36f36 WatchSource:0}: Error finding container d7b2d290acd75f87632518d71155e48c58ea0cec4d1f9b270bc9bb1601f36f36: Status 404 returned error can't find the container with id d7b2d290acd75f87632518d71155e48c58ea0cec4d1f9b270bc9bb1601f36f36 Oct 31 03:09:29.083786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.083750 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:29.083889 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.083830 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:29.083889 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.083876 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:29.190601 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.190560 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:29.191316 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.191168 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:29.191316 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.191228 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:29.191562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.190791 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5"] Oct 31 03:09:29.631206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.631069 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" event={"ID":"7d087ee5-7714-4cc3-88a8-de6e13297913","Type":"ContainerStarted","Data":"bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085"} Oct 31 03:09:29.631206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.631170 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" event={"ID":"7d087ee5-7714-4cc3-88a8-de6e13297913","Type":"ContainerStarted","Data":"d7b2d290acd75f87632518d71155e48c58ea0cec4d1f9b270bc9bb1601f36f36"} Oct 31 03:09:29.633441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.633413 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" event={"ID":"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5","Type":"ContainerStarted","Data":"312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc"} Oct 31 03:09:29.633600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.633456 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" event={"ID":"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5","Type":"ContainerStarted","Data":"3b21c758f78f06e7d306a43f6f5da2806a65f322f33cabdf17fbd900b08845b7"} Oct 31 03:09:29.636436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.636408 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" event={"ID":"4f59a4f5-d136-4c9b-a557-792bb6b71657","Type":"ContainerStarted","Data":"d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e"} Oct 31 03:09:29.636528 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.636457 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" event={"ID":"4f59a4f5-d136-4c9b-a557-792bb6b71657","Type":"ContainerStarted","Data":"63d8c35befae4b44ebab085ded9b8db8eb334fddcda2c66aec27df941320e83d"} Oct 31 03:09:29.678062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.671369 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:29.690106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.689852 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" podStartSLOduration=1.68982788 podStartE2EDuration="1.68982788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:29.671135118 +0000 UTC m=+851.351661248" watchObservedRunningTime="2026-10-31 03:09:29.68982788 +0000 UTC m=+851.370354009" Oct 31 03:09:29.752059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.736022 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" podStartSLOduration=1.73600126 podStartE2EDuration="1.73600126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:29.732765834 +0000 UTC m=+851.413291962" watchObservedRunningTime="2026-10-31 03:09:29.73600126 +0000 UTC m=+851.416527388" Oct 31 03:09:29.752059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.740920 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" podStartSLOduration=1.740902948 podStartE2EDuration="1.740902948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:29.704410243 +0000 UTC m=+851.384936371" watchObservedRunningTime="2026-10-31 03:09:29.740902948 +0000 UTC m=+851.421429075" Oct 31 03:09:29.772142 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.772108 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:29.878196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:29.878160 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:30.667062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:30.665485 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:30.667062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:30.665652 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:09:30.776081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:30.773446 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:30.776356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:30.776322 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 03:09:30.886066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:30.885203 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:09:30.886066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:30.885314 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:09:31.659452 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:31.659413 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:31.659705 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:31.659676 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:09:31.776987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:31.776949 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:31.779325 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:31.779294 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 03:09:31.876521 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:31.876481 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:09:31.876785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:31.876747 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:09:32.208080 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:09:32.200590 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:09:32.656117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:32.655611 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:32.656117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:32.655702 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:09:32.787059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:32.785462 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:32.787059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:32.785535 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 03:09:32.877533 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:32.877498 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:09:32.877844 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:32.877799 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:09:33.666062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:33.665459 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:33.666062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:33.665524 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:09:33.774078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:33.773263 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:33.774078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:33.773332 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 03:09:33.878062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:33.877669 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:09:33.878062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:33.877727 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:09:34.655599 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:34.655559 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:34.655902 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:34.655870 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:09:34.773410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:34.773372 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:34.773669 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:34.773646 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 03:09:34.877554 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:34.877506 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:09:34.877732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:34.877580 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:09:35.655067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:35.654532 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:35.655067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:35.654610 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:09:35.775061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:35.774129 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:35.775061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:35.774260 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 03:09:35.876470 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:35.876428 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:09:35.876663 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:35.876497 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:09:36.676948 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:36.676912 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:36.677783 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:36.677759 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:36.678949 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:36.678927 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:09:36.796029 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:36.796001 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:36.797611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:36.797589 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:36.798692 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:36.798675 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:09:37.023064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:37.022952 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:37.023494 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:37.023472 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:37.024297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:37.024272 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:09:41.473413 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.473376 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts"] Oct 31 03:09:41.477626 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.477601 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.490453 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.490417 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 03:09:41.490628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.490611 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-97phk\"" Oct 31 03:09:41.490718 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.490498 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 03:09:41.490718 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.490476 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 03:09:41.495431 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.495412 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts"] Oct 31 03:09:41.567996 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.567953 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9rdn\" (UniqueName: \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-kube-api-access-t9rdn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.568234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.568099 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.568234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.568219 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.568361 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.568311 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabaf37d-047e-4a35-aa52-646cd02c3496-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.568361 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.568353 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.568465 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.568415 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.568465 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.568439 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.669466 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.669431 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.669668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.669471 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.669668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.669575 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9rdn\" (UniqueName: \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-kube-api-access-t9rdn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.669788 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.669669 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.669788 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.669747 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.669897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.669812 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabaf37d-047e-4a35-aa52-646cd02c3496-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.669897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.669868 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.670723 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.670692 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.672682 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.672659 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabaf37d-047e-4a35-aa52-646cd02c3496-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.672831 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.672655 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.672894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.672854 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.673549 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.673525 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.682691 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.682658 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.682926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.682904 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9rdn\" (UniqueName: \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-kube-api-access-t9rdn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cnfts\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:41.801737 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:41.801640 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:42.087221 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.087190 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts"] Oct 31 03:09:42.087716 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:09:42.087682 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabaf37d_047e_4a35_aa52_646cd02c3496.slice/crio-9a32ed62a2b1be9a4968a8b8b097af1add200160601c15437d4cad50d5dc1cb5 WatchSource:0}: Error finding container 9a32ed62a2b1be9a4968a8b8b097af1add200160601c15437d4cad50d5dc1cb5: Status 404 returned error can't find the container with id 9a32ed62a2b1be9a4968a8b8b097af1add200160601c15437d4cad50d5dc1cb5 Oct 31 03:09:42.795472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.795438 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv"] Oct 31 03:09:42.801613 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.801586 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:42.815751 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.815669 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv"] Oct 31 03:09:42.961152 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.961117 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" event={"ID":"cabaf37d-047e-4a35-aa52-646cd02c3496","Type":"ContainerStarted","Data":"9a32ed62a2b1be9a4968a8b8b097af1add200160601c15437d4cad50d5dc1cb5"} Oct 31 03:09:42.987350 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.987317 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:42.987552 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.987362 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:42.987617 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.987566 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:42.987681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.987615 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:42.987873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.987834 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:42.987940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.987878 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:42.987940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.987913 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:42.988062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.987965 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qlsr\" (UniqueName: \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-kube-api-access-7qlsr\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:42.988062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:42.987995 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.054195 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.054093 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57"] Oct 31 03:09:43.058827 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.058802 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.086871 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.084323 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57"] Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.089877 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.089922 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.089950 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.089978 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qlsr\" (UniqueName: \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-kube-api-access-7qlsr\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090010 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090114 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqtw4\" (UniqueName: \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-kube-api-access-nqtw4\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090162 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090196 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090233 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090278 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090329 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090394 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090429 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090463 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090527 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090553 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8cbfb8-a275-4705-9cea-441b590f2436-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090591 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.090626 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.092722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.092386 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.096025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.095027 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.096025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.095300 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.096025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.095793 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.096025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.095989 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.108934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.108882 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.112182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.112146 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.124975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.115546 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.125322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.116154 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qlsr\" (UniqueName: \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-kube-api-access-7qlsr\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:43.192115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192072 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.192115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192118 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8cbfb8-a275-4705-9cea-441b590f2436-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.192115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192169 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.192469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192205 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.192469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192306 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtw4\" (UniqueName: \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-kube-api-access-nqtw4\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.192469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192364 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.192469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192410 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.192684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192456 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.192684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192574 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.192684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192637 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.192965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.192920 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.193468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.193356 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8cbfb8-a275-4705-9cea-441b590f2436-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.193468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.193383 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.193468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.193426 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.194873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.194848 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.197163 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.197138 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.207079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.207014 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtw4\" (UniqueName: \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-kube-api-access-nqtw4\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.210511 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.210473 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.381996 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.381914 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:43.424977 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:43.424926 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:44.192077 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:44.166405 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57"] Oct 31 03:09:44.277066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:44.274999 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:44.277066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:44.275190 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:44.277066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:44.275312 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:45.110626 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.110219 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:09:45.140270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.140209 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:09:45.317112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.315955 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" event={"ID":"5e8cbfb8-a275-4705-9cea-441b590f2436","Type":"ContainerStarted","Data":"91c90a193dab0d999fd18f092c450a63501389799ccccaf557a00c063e259b2e"} Oct 31 03:09:45.405376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.402398 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:45.430099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.430014 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" podStartSLOduration=2.429992337 podStartE2EDuration="2.429992337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:45.357735032 +0000 UTC m=+867.038261173" watchObservedRunningTime="2026-10-31 03:09:45.429992337 +0000 UTC m=+867.110518465" Oct 31 03:09:45.440093 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:09:45.435341 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f9a80b_b1f2_46e7_a9c8_2cbde2b47485.slice/crio-c94498c3639f71742aed0b1fec7f38529dd9d6223b3d495d94575c3d5e14d186 WatchSource:0}: Error finding container c94498c3639f71742aed0b1fec7f38529dd9d6223b3d495d94575c3d5e14d186: Status 404 returned error can't find the container with id c94498c3639f71742aed0b1fec7f38529dd9d6223b3d495d94575c3d5e14d186 Oct 31 03:09:45.444081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.441398 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:45.444081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.441494 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:45.444081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.441541 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:45.512195 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.502516 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv"] Oct 31 03:09:45.568066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.567058 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:09:45.612664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:45.612463 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:09:46.362803 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:46.351585 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" event={"ID":"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485","Type":"ContainerStarted","Data":"ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c"} Oct 31 03:09:46.362803 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:46.351695 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" event={"ID":"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485","Type":"ContainerStarted","Data":"c94498c3639f71742aed0b1fec7f38529dd9d6223b3d495d94575c3d5e14d186"} Oct 31 03:09:46.362803 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:46.361437 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" event={"ID":"5e8cbfb8-a275-4705-9cea-441b590f2436","Type":"ContainerStarted","Data":"6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a"} Oct 31 03:09:46.397916 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:46.397884 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:46.398731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:46.398690 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:09:46.430006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:46.429971 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:46.430351 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:46.430326 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:09:46.430526 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:46.430499 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 03:09:47.400002 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:47.398489 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:47.400002 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:47.398553 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:09:47.425989 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:47.425656 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:09:47.425989 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:47.425710 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 03:09:48.393101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:48.391108 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:48.393101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:48.391168 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:09:48.425508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:48.425463 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:09:48.425978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:48.425521 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 03:09:49.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:49.391444 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:49.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:49.391506 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:09:49.427055 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:49.426327 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:09:49.427055 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:49.426373 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 03:09:50.391061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:50.390438 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:50.391061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:50.390501 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:09:50.456060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:50.450173 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:09:50.456060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:50.450236 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 03:09:51.383061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:51.382410 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:51.383061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:51.382479 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:09:51.426059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:51.425290 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:09:51.426059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:51.425355 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 03:09:52.382538 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:52.382497 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:52.382965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:52.382564 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:09:52.425955 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:52.425517 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:09:52.425955 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:52.425577 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 03:09:53.495064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:53.494387 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:53.496387 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:53.495662 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:53.498641 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:53.498621 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:09:53.524364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:53.524312 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" podStartSLOduration=11.524297694 podStartE2EDuration="11.524297694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:46.39934327 +0000 UTC m=+868.079869400" watchObservedRunningTime="2026-10-31 03:09:53.524297694 +0000 UTC m=+875.204823824" Oct 31 03:09:53.534152 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:53.534111 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:53.534238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:53.534199 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:09:53.696766 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:53.696730 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:53.697178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:53.697154 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:53.699152 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:53.699120 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:09:54.602497 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:54.602461 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" event={"ID":"cabaf37d-047e-4a35-aa52-646cd02c3496","Type":"ContainerStarted","Data":"40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8"} Oct 31 03:09:54.602991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:54.602522 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:09:54.604734 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:54.604711 4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-cnfts container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:09:54.604845 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:54.604755 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" podUID="cabaf37d-047e-4a35-aa52-646cd02c3496" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:09:54.640422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:54.640332 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" podStartSLOduration=2.197690134 podStartE2EDuration="13.640315265s" totalImagesPullingTime="11.442625131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:54.632834461 +0000 UTC m=+876.313360589" watchObservedRunningTime="2026-10-31 03:09:54.640315265 +0000 UTC m=+876.320841393" Oct 31 03:09:55.610779 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:09:55.610744 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:10:02.409253 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:10:02.409216 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:10:03.446246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:03.445543 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:10:03.552214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:03.552142 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:03.774945 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:03.774839 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:13.000553 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.000518 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc"] Oct 31 03:10:13.005288 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.005268 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.020054 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.020020 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc"] Oct 31 03:10:13.120493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.119553 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.120493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.119603 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8ce63a-cace-42b0-891c-836915b09ca7-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.120493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.119703 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.120493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.119778 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.120493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.119818 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.122317 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.119869 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gx2z\" (UniqueName: \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-kube-api-access-7gx2z\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.124493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.123295 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.124493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.123470 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.124493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.123529 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.225311 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.225273 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.225501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.225323 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8ce63a-cace-42b0-891c-836915b09ca7-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.225501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.225399 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.225501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.225452 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.225501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.225480 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.225734 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.225664 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gx2z\" (UniqueName: \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-kube-api-access-7gx2z\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.225734 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.225707 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.225848 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.225768 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.225848 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.225821 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.226198 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.226173 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8ce63a-cace-42b0-891c-836915b09ca7-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.226296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.226225 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.226296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.226228 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.226517 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.226497 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.226587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.226542 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.228101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.228081 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.228475 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.228457 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.239881 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.239857 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gx2z\" (UniqueName: \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-kube-api-access-7gx2z\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.239979 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.239956 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.321913 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.321833 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:13.554950 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.554910 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z"] Oct 31 03:10:13.560156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.560130 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.582201 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.582130 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z"] Oct 31 03:10:13.633757 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:10:13.633722 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8ce63a_cace_42b0_891c_836915b09ca7.slice/crio-9cf77f54aff1c38049092d6067f93dcb905557d48376182f904e3d9384303048 WatchSource:0}: Error finding container 9cf77f54aff1c38049092d6067f93dcb905557d48376182f904e3d9384303048: Status 404 returned error can't find the container with id 9cf77f54aff1c38049092d6067f93dcb905557d48376182f904e3d9384303048 Oct 31 03:10:13.635399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.635368 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc"] Oct 31 03:10:13.637087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.637019 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:13.637171 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.637149 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:13.637230 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.637193 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:13.735684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.735649 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.735874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.735691 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe931dea-f20e-4671-9eed-2d800466d2f0-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.735874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.735830 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.735874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.735867 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.736084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.735896 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.736084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.735977 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.736181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.736158 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.736229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.736196 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s954\" (UniqueName: \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-kube-api-access-2s954\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.736356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.736336 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.817839 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.817798 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" event={"ID":"aa8ce63a-cace-42b0-891c-836915b09ca7","Type":"ContainerStarted","Data":"1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110"} Oct 31 03:10:13.818075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.817868 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" event={"ID":"aa8ce63a-cace-42b0-891c-836915b09ca7","Type":"ContainerStarted","Data":"9cf77f54aff1c38049092d6067f93dcb905557d48376182f904e3d9384303048"} Oct 31 03:10:13.837793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.837713 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.837793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.837752 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s954\" (UniqueName: \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-kube-api-access-2s954\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.837814 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.837942 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.837989 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe931dea-f20e-4671-9eed-2d800466d2f0-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.838084 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.838113 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.838140 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.838182 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.838216 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838566 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.838384 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838566 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.838547 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.838672 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.838643 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.839500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.839472 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe931dea-f20e-4671-9eed-2d800466d2f0-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.841077 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.840993 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.841272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.841252 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.848350 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.848291 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" podStartSLOduration=1.848273253 podStartE2EDuration="1.848273253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:13.845859899 +0000 UTC m=+895.526386025" watchObservedRunningTime="2026-10-31 03:10:13.848273253 +0000 UTC m=+895.528799388" Oct 31 03:10:13.852726 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.852701 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.854289 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.854264 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s954\" (UniqueName: \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-kube-api-access-2s954\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:13.897033 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:13.896995 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:14.239421 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.235948 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z"] Oct 31 03:10:14.239421 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:10:14.237530 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe931dea_f20e_4671_9eed_2d800466d2f0.slice/crio-a91d58321a50d2828d142be04f911e5afc26c1fe3460104df02e14bb235ba142 WatchSource:0}: Error finding container a91d58321a50d2828d142be04f911e5afc26c1fe3460104df02e14bb235ba142: Status 404 returned error can't find the container with id a91d58321a50d2828d142be04f911e5afc26c1fe3460104df02e14bb235ba142 Oct 31 03:10:14.244614 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.244563 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:14.244858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.244834 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:14.245020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.244994 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:14.324624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.324138 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:14.324624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.324284 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:10:14.324624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.324331 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:10:14.830514 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.830467 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" event={"ID":"fe931dea-f20e-4671-9eed-2d800466d2f0","Type":"ContainerStarted","Data":"9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c"} Oct 31 03:10:14.830944 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.830798 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" event={"ID":"fe931dea-f20e-4671-9eed-2d800466d2f0","Type":"ContainerStarted","Data":"a91d58321a50d2828d142be04f911e5afc26c1fe3460104df02e14bb235ba142"} Oct 31 03:10:14.869838 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.869769 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" podStartSLOduration=1.869746894 podStartE2EDuration="1.869746894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:14.860523198 +0000 UTC m=+896.541049337" watchObservedRunningTime="2026-10-31 03:10:14.869746894 +0000 UTC m=+896.550273024" Oct 31 03:10:14.897930 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:14.897897 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:15.326063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:15.325816 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:10:15.326655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:15.325901 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:10:15.907058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:15.906202 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:15.907058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:15.906265 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:10:16.322430 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:16.322346 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:10:16.322430 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:16.322403 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:10:16.905061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:16.897425 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:16.905061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:16.897488 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:10:17.323110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:17.322988 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:10:17.323110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:17.323075 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:10:17.915064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:17.905592 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:17.915064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:17.905657 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:10:18.325134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:18.322632 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:10:18.325134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:18.322685 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:10:18.898060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:18.897790 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:18.898060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:18.897865 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:10:19.323232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:19.323191 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:10:19.323695 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:19.323670 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:10:19.898481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:19.898444 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:19.898775 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:19.898750 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:10:20.322884 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:20.322851 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:10:20.323134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:20.323108 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:10:20.897780 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:20.897739 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:20.898228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:20.897833 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:10:21.601639 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:21.601597 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:21.602200 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:21.602167 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:21.602920 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:21.602901 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:10:22.064904 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:22.064847 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:22.066144 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:22.066120 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:22.067175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:22.067136 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:10:27.882208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:27.882167 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8"] Oct 31 03:10:27.902002 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:27.901960 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:27.908448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:27.908417 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8"] Oct 31 03:10:28.002252 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.002215 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.002475 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.002457 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.002566 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.002489 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31b4da9-4b9f-4ff2-85be-343786144bf8-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.002617 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.002593 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.002658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.002640 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.002706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.002688 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.002783 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.002767 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz4xn\" (UniqueName: \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-kube-api-access-lz4xn\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.002817 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.002804 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.002861 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.002824 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104169 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104131 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104169 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104168 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31b4da9-4b9f-4ff2-85be-343786144bf8-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104198 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104216 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104233 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104274 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz4xn\" (UniqueName: \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-kube-api-access-lz4xn\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104301 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104329 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104372 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104631 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.104987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.104960 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.105138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.105076 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.105301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.105269 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.105396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.105357 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31b4da9-4b9f-4ff2-85be-343786144bf8-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.106973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.106947 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.107178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.107159 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.126281 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.126252 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz4xn\" (UniqueName: \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-kube-api-access-lz4xn\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.127216 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.127195 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.177116 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.177081 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb"] Oct 31 03:10:28.184577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.184546 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.201678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.201649 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb"] Oct 31 03:10:28.242385 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.242352 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:28.312676 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.312640 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.313100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.312884 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.313431 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.313327 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.313431 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.313379 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.313536 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.313451 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.313633 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.313582 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.314020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.313614 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.314020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.313740 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhxwx\" (UniqueName: \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-kube-api-access-nhxwx\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.314020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.313773 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.414639 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.414601 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.414827 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.414730 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.414827 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.414755 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.414827 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.414780 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhxwx\" (UniqueName: \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-kube-api-access-nhxwx\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.414827 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.414809 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.414967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.414862 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.414967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.414887 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.415090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.415071 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.415132 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.415118 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.415532 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.415503 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.415681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.415631 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.415888 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.415865 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.415991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.415971 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.416188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.416170 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.417328 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.417306 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.417948 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.417930 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.428355 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.428293 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhxwx\" (UniqueName: \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-kube-api-access-nhxwx\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.432817 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.432788 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.506206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.506162 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:28.550954 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.550896 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8"] Oct 31 03:10:28.554534 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:10:28.554502 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31b4da9_4b9f_4ff2_85be_343786144bf8.slice/crio-4733deed2ac5f39c118b7aa2ce50c9f0a70180abc71b886fff757f64be90aeb4 WatchSource:0}: Error finding container 4733deed2ac5f39c118b7aa2ce50c9f0a70180abc71b886fff757f64be90aeb4: Status 404 returned error can't find the container with id 4733deed2ac5f39c118b7aa2ce50c9f0a70180abc71b886fff757f64be90aeb4 Oct 31 03:10:28.558859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.558132 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:28.558859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.558225 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:28.558859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.558264 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:28.846970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.846934 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb"] Oct 31 03:10:28.871625 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:10:28.870131 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99b5e71_0870_42ed_bd1a_302562ecaa8c.slice/crio-72d5af275b0eca7a9efd0c6d1add010288904d200ca17e694e4cb2fa2cd0d37f WatchSource:0}: Error finding container 72d5af275b0eca7a9efd0c6d1add010288904d200ca17e694e4cb2fa2cd0d37f: Status 404 returned error can't find the container with id 72d5af275b0eca7a9efd0c6d1add010288904d200ca17e694e4cb2fa2cd0d37f Oct 31 03:10:28.900521 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.898058 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:28.900521 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.898156 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:28.900521 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:28.898200 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:29.008533 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:29.008492 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" event={"ID":"b99b5e71-0870-42ed-bd1a-302562ecaa8c","Type":"ContainerStarted","Data":"72d5af275b0eca7a9efd0c6d1add010288904d200ca17e694e4cb2fa2cd0d37f"} Oct 31 03:10:29.012303 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:29.012272 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" event={"ID":"d31b4da9-4b9f-4ff2-85be-343786144bf8","Type":"ContainerStarted","Data":"1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e"} Oct 31 03:10:29.012495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:29.012480 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" event={"ID":"d31b4da9-4b9f-4ff2-85be-343786144bf8","Type":"ContainerStarted","Data":"4733deed2ac5f39c118b7aa2ce50c9f0a70180abc71b886fff757f64be90aeb4"} Oct 31 03:10:29.248155 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:29.245785 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:29.248155 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:29.245914 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:29.248155 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:29.245957 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:10:30.112296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:30.093117 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" event={"ID":"b99b5e71-0870-42ed-bd1a-302562ecaa8c","Type":"ContainerStarted","Data":"d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525"} Oct 31 03:10:30.152245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:30.147696 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podStartSLOduration=2.147668532 podStartE2EDuration="2.147668532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:30.145515272 +0000 UTC m=+911.826041413" watchObservedRunningTime="2026-10-31 03:10:30.147668532 +0000 UTC m=+911.828194661" Oct 31 03:10:30.152245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:30.148138 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podStartSLOduration=3.148127886 podStartE2EDuration="3.148127886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:29.052118552 +0000 UTC m=+910.732644679" watchObservedRunningTime="2026-10-31 03:10:30.148127886 +0000 UTC m=+911.828654016" Oct 31 03:10:30.253483 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:30.253447 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:30.253779 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:30.253749 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:10:30.509063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:30.508890 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:30.509063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:30.509008 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:10:30.509321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:30.509119 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:10:30.638298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:30.637226 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.302064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:31.270415 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:31.302064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:31.270477 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:10:31.522525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:31.522483 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:10:31.522792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:31.522765 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:10:31.641297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:31.641185 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.643683 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:31.643662 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:10:31.803285 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:31.803236 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.888583 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:31.888549 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:10:32.034069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:32.030026 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 03:10:32.071882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:32.071814 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.337740 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:32.337641 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:32.363183 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:32.363137 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:10:32.564178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:32.554976 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:10:32.651544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:32.651490 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:10:32.795075 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:10:32.795018 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:10:32.944240 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:32.944137 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.034552 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.034505 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 03:10:33.047215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.047179 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:10:33.080317 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.080272 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:10:33.087215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.081257 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:10:33.162131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.160180 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:10:33.263123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.260094 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:33.263123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.260163 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:10:33.430007 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.396521 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.466061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.439799 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.517123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.512834 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:10:33.517404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.517371 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:10:33.580257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.560180 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 03:10:33.703004 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.702973 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:10:33.974065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.951130 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.974065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:33.951606 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:10:34.038302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:34.038261 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:34.090319 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:34.067425 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:10:34.241820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:34.241552 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:10:34.243855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:34.242768 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:34.243855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:34.243526 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:10:34.533116 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:34.531561 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:10:34.533116 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:34.531636 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:10:35.242733 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:35.242701 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:35.242894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:35.242754 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:10:35.507117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:35.506855 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:10:35.507117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:35.506918 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:10:36.243102 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:36.243032 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:36.243479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:36.243129 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:10:36.514351 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:36.514256 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:10:36.514606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:36.514571 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:10:37.243426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:37.243393 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:37.243895 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:37.243867 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:10:37.508399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:37.508285 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:10:37.508653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:37.508622 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:10:38.243024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:38.242952 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:38.243024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:38.243012 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:10:38.507815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:38.507726 4456 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:10:38.507815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:38.507791 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:10:39.327575 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:39.327529 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:39.330648 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:39.329396 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:39.333445 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:39.333418 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:10:39.521475 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:39.521442 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:39.522867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:39.522843 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:39.523794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:39.523772 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:10:43.318402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.318354 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5"] Oct 31 03:10:43.322975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.322952 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.332947 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.332926 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-85cb59-openshift-default-dockercfg-fr8qt\"" Oct 31 03:10:43.343365 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.343338 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5"] Oct 31 03:10:43.420854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.420812 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.421087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.420877 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.421087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.420979 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.421087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.421018 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.421257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.421103 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.421257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.421166 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.421257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.421198 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.421417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.421297 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j592\" (UniqueName: \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-kube-api-access-9j592\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.421417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.421336 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.522924 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.522887 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.522979 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523010 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523071 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523101 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523139 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523176 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523206 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523274 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j592\" (UniqueName: \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-kube-api-access-9j592\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523617 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523451 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523717 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.523897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523873 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.524013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.523992 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.524181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.524165 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.525819 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.525794 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.526260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.526241 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.537163 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.537115 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.537301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.537278 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j592\" (UniqueName: \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-kube-api-access-9j592\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.640883 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.640795 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:43.933459 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:10:43.933421 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635cf9bc_77c2_4809_9e6a_1f9d331431bb.slice/crio-b083eddd105a225b84a02f3caf679158e2895fff7ce97d7378222a148b808950 WatchSource:0}: Error finding container b083eddd105a225b84a02f3caf679158e2895fff7ce97d7378222a148b808950: Status 404 returned error can't find the container with id b083eddd105a225b84a02f3caf679158e2895fff7ce97d7378222a148b808950 Oct 31 03:10:43.933821 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.933797 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5"] Oct 31 03:10:43.936110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.936031 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:43.936203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.936181 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:43.936271 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:43.936218 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:44.835727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:44.835663 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" event={"ID":"635cf9bc-77c2-4809-9e6a-1f9d331431bb","Type":"ContainerStarted","Data":"428aabd3bda49291197b74e2dc70a3ecd615dfcfc45f3734d342995c02464a38"} Oct 31 03:10:44.836373 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:44.835800 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" event={"ID":"635cf9bc-77c2-4809-9e6a-1f9d331431bb","Type":"ContainerStarted","Data":"b083eddd105a225b84a02f3caf679158e2895fff7ce97d7378222a148b808950"} Oct 31 03:10:44.876846 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:44.876760 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" podStartSLOduration=1.876739084 podStartE2EDuration="1.876739084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:44.863533395 +0000 UTC m=+926.544059533" watchObservedRunningTime="2026-10-31 03:10:44.876739084 +0000 UTC m=+926.557265211" Oct 31 03:10:45.642011 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:45.641487 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:45.643087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:45.643054 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:10:45.643274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:45.643135 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 03:10:46.641651 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:46.641615 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:10:46.642052 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:46.641668 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 03:10:47.641997 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:47.641961 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:10:47.642400 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:47.642015 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 03:10:48.641423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:48.641387 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:10:48.641607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:48.641452 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 03:10:49.642193 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:49.642151 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:10:49.642778 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:49.642748 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 03:10:50.641607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:50.641570 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:10:50.641818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:50.641627 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 03:10:51.645970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:51.645940 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:51.646481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:51.646464 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:51.647236 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:51.647217 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:10:58.101967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.101926 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc"] Oct 31 03:10:58.108332 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.108301 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.123497 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.123469 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc"] Oct 31 03:10:58.124712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.124311 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.124712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.124385 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9lb\" (UniqueName: \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-kube-api-access-pm9lb\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.124712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.124474 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.124712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.124530 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.124712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.124558 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.124712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.124619 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.127487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.124789 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.127487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.124824 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.127487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.124851 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.225678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.225639 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.225874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.225685 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.225874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.225762 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226009 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.225954 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226009 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.225989 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226148 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.226019 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226148 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.226087 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226249 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.226142 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226249 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.226147 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9lb\" (UniqueName: \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-kube-api-access-pm9lb\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226360 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.226266 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226360 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.226311 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226360 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.226342 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226553 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.226535 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.226820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.226771 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.228460 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.228437 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.228576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.228506 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.256323 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.256245 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.259632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.259605 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9lb\" (UniqueName: \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-kube-api-access-pm9lb\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.303791 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.303736 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl"] Oct 31 03:10:58.311454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.311415 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.328100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.328068 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.328249 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.328169 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.328249 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.328214 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.328337 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.328251 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.328337 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.328288 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-976fq\" (UniqueName: \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-kube-api-access-976fq\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.328417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.328381 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.328490 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.328467 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.328534 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.328503 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.328649 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.328627 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.330724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.330698 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl"] Oct 31 03:10:58.429872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.429831 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.430209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.430182 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.430209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.430219 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.430517 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.430341 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.430517 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.430404 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.430517 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.430489 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.430662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.430525 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.430662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.430558 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.430662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.430588 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976fq\" (UniqueName: \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-kube-api-access-976fq\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.431699 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.431669 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.431970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.431948 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.432212 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.432191 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.432464 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.432432 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.433414 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.433377 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.435057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.435014 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.437269 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.437244 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.444743 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.444718 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-976fq\" (UniqueName: \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-kube-api-access-976fq\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.446273 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.446251 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:58.449006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.448213 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.633276 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.633237 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:10:58.750924 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.750421 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc"] Oct 31 03:10:58.753875 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:10:58.753845 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e5f3ec_ce41_47bd_8d75_bce776c5a8c3.slice/crio-18d1ee6d8e277e45c718085db9de4f7d863cd11482fdae7dd701ae560c13cc73 WatchSource:0}: Error finding container 18d1ee6d8e277e45c718085db9de4f7d863cd11482fdae7dd701ae560c13cc73: Status 404 returned error can't find the container with id 18d1ee6d8e277e45c718085db9de4f7d863cd11482fdae7dd701ae560c13cc73 Oct 31 03:10:58.756628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.756590 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:58.756725 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.756663 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:58.756725 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.756694 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:58.935056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.934892 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl"] Oct 31 03:10:58.940387 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:10:58.940354 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc17f71_5f77_48e0_a0c9_395bb51b4388.slice/crio-3095badd1b5504057ad7434b5e1bbbf4698595fdc1b532a94e853c76fd5d02b9 WatchSource:0}: Error finding container 3095badd1b5504057ad7434b5e1bbbf4698595fdc1b532a94e853c76fd5d02b9: Status 404 returned error can't find the container with id 3095badd1b5504057ad7434b5e1bbbf4698595fdc1b532a94e853c76fd5d02b9 Oct 31 03:10:58.943181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.943020 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:58.943256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.943214 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:58.943298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.943260 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:10:58.992331 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.992263 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" event={"ID":"cfc17f71-5f77-48e0-a0c9-395bb51b4388","Type":"ContainerStarted","Data":"3095badd1b5504057ad7434b5e1bbbf4698595fdc1b532a94e853c76fd5d02b9"} Oct 31 03:10:58.994594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.994563 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" event={"ID":"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3","Type":"ContainerStarted","Data":"08f03631154d59f0f7f994e61019e9139c04fb20c85b86bd50a5c717bfea7a85"} Oct 31 03:10:58.994695 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:58.994617 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" event={"ID":"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3","Type":"ContainerStarted","Data":"18d1ee6d8e277e45c718085db9de4f7d863cd11482fdae7dd701ae560c13cc73"} Oct 31 03:10:59.062062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:59.045316 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podStartSLOduration=1.045296381 podStartE2EDuration="1.045296381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:59.030372523 +0000 UTC m=+940.710898653" watchObservedRunningTime="2026-10-31 03:10:59.045296381 +0000 UTC m=+940.725822508" Oct 31 03:10:59.453872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:59.453837 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:10:59.455803 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:59.455775 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:10:59.456017 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:10:59.455994 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:00.071062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:00.070286 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" event={"ID":"cfc17f71-5f77-48e0-a0c9-395bb51b4388","Type":"ContainerStarted","Data":"594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78"} Oct 31 03:11:00.143057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:00.142735 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podStartSLOduration=2.142713197 podStartE2EDuration="2.142713197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:00.108451284 +0000 UTC m=+941.788977413" watchObservedRunningTime="2026-10-31 03:11:00.142713197 +0000 UTC m=+941.823239325" Oct 31 03:11:00.501061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:00.494420 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:00.501061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:00.494479 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:00.643086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:00.639658 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:11:00.643086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:00.639825 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:00.643086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:00.639867 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:11:01.279061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:01.277187 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.439341 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:01.427160 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.498075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:01.492070 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:01.498075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:01.492147 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:01.662138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:01.640681 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:01.662138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:01.640755 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:11:02.223062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.220777 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 03:11:02.248062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.235141 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.383094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.261169 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:11:02.383094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.261368 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.383094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.363654 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.383094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.365155 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:11:02.431798 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.431063 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.431798 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.431657 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:11:02.600502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.600328 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.663595 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.635447 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Oct 31 03:11:02.765054 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.765003 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:02.765388 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.765335 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:11:02.811778 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.796187 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:11:02.814245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.814204 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:02.814511 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:02.814475 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:03.115068 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:11:03.088536 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:11:03.177670 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.177629 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:11:03.231068 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.222511 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.253176 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.253143 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:11:03.328091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.309631 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:11:03.335607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.326378 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Oct 31 03:11:03.355880 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.355849 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:11:03.357550 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.326427 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.408738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.408708 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:11:03.451335 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.451302 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:03.451637 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.451592 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:03.659229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.553151 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.659684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.604172 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.714235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.714194 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:03.771067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.748156 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:11:03.835063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.800169 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.835063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:03.832146 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:11:04.038277 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.038172 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 31 03:11:04.234193 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.234139 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.334343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.334241 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 31 03:11:04.340716 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.340330 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:11:04.459167 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.432658 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.461447 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.460842 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:04.461447 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.460896 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:04.617087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.615533 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.650333 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.650296 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:04.651087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.651029 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:11:04.662184 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.662147 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 03:11:04.662750 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.662707 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:11:04.664816 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.664788 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.895178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:04.883452 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:11:05.271062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.259932 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 03:11:05.343188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.343142 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.423113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.420567 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:11:05.468419 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.466379 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:05.468419 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.466455 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:05.634781 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.615845 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.634781 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.616535 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:11:05.652099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.635822 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:05.652099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.635879 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:11:05.664269 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.664238 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:11:05.675889 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.675840 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.680076 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.676210 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.928865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:05.928682 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:11:06.026960 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:06.026449 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:11:06.239414 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:06.234873 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:11:06.245389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:06.245361 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:11:06.387059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:06.360497 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:06.477189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:06.477141 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:06.477527 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:06.477493 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:06.544810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:06.544726 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:11:06.633884 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:06.633849 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:06.634092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:06.633911 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:11:07.457073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:07.456663 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:07.457073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:07.456722 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:07.635113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:07.634625 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:07.635113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:07.634688 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:11:08.456090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:08.454392 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:08.456090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:08.454458 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:08.633742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:08.633707 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:08.634259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:08.633792 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:11:09.470860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:09.465853 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:09.470860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:09.465912 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:11:09.633828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:09.633787 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:09.634268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:09.633847 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:11:11.110476 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:11.110433 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:11:11.111714 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:11.111690 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:11:11.114475 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:11.114436 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:11:11.496261 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:11.496194 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:11.496486 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:11.496307 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:11.837202 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:11.837121 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:11:11.837599 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:11.837572 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:11:11.838505 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:11.838482 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:11:28.462256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.462211 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs"] Oct 31 03:11:28.484337 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.484306 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs"] Oct 31 03:11:28.484544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.484404 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.500089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.500056 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97"] Oct 31 03:11:28.510112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.507911 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.516961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.516917 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97"] Oct 31 03:11:28.571100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.571030 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gzgr\" (UniqueName: \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-kube-api-access-4gzgr\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.571278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.571205 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.571322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.571279 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.571416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.571390 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.571484 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.571449 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.571610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.571596 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.571647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.571632 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.571684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.571657 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.571773 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.571752 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.673577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.673542 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.673792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.673603 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8fd\" (UniqueName: \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-kube-api-access-tj8fd\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.673792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.673710 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.673792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.673759 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.673792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.673785 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.673809 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.673844 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.674025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.673897 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gzgr\" (UniqueName: \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-kube-api-access-4gzgr\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.673935 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.674025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.673973 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1605d8d-256f-42f5-b3fc-448e3dde1950-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.674327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674067 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674130 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.674327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674193 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674216 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.674327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674242 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674305 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674424 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674460 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.674624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674504 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674555 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.674624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674587 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674883 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674830 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.674978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.674957 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.676738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.676712 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.677350 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.677321 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.688139 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.688085 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.690302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.689992 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gzgr\" (UniqueName: \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-kube-api-access-4gzgr\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.775828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.775736 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.775828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.775803 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.776079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.775865 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.776079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.775933 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.776079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.775998 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.776267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.776079 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8fd\" (UniqueName: \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-kube-api-access-tj8fd\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.776267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.776172 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.776267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.776223 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.776267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.776263 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1605d8d-256f-42f5-b3fc-448e3dde1950-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.776472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.776307 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.776585 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.776556 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.776845 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.776825 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.777135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.777111 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1605d8d-256f-42f5-b3fc-448e3dde1950-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.777447 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.777427 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.778767 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.778739 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.780308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.780284 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.790078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.790003 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.790279 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.790260 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8fd\" (UniqueName: \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-kube-api-access-tj8fd\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:28.815804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.815767 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:28.833395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:28.833363 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:29.113283 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.113252 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs"] Oct 31 03:11:29.113388 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:11:29.113357 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5e43a2_a261_4c0a_8ad6_34a5774497e4.slice/crio-a013d389e2ab91002584ac0be1dac28d87d5422137c603dc26a04637e2ec6035 WatchSource:0}: Error finding container a013d389e2ab91002584ac0be1dac28d87d5422137c603dc26a04637e2ec6035: Status 404 returned error can't find the container with id a013d389e2ab91002584ac0be1dac28d87d5422137c603dc26a04637e2ec6035 Oct 31 03:11:29.116372 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.116331 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:11:29.116482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.116401 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:11:29.116482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.116429 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:11:29.167597 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:11:29.167565 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1605d8d_256f_42f5_b3fc_448e3dde1950.slice/crio-aa5aaf65280fcb2b4e1143267b30b11c020fca9a6dfc7956c30ae0a57c2eff16 WatchSource:0}: Error finding container aa5aaf65280fcb2b4e1143267b30b11c020fca9a6dfc7956c30ae0a57c2eff16: Status 404 returned error can't find the container with id aa5aaf65280fcb2b4e1143267b30b11c020fca9a6dfc7956c30ae0a57c2eff16 Oct 31 03:11:29.168866 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.168836 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97"] Oct 31 03:11:29.170668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.170634 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:11:29.170757 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.170698 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:11:29.170757 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.170728 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:11:29.568631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.567446 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" event={"ID":"7a5e43a2-a261-4c0a-8ad6-34a5774497e4","Type":"ContainerStarted","Data":"989312fdbf5b95b023c1150e973e778863d119ee8708b246f10117cd4bcc034d"} Oct 31 03:11:29.568631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.567520 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" event={"ID":"7a5e43a2-a261-4c0a-8ad6-34a5774497e4","Type":"ContainerStarted","Data":"a013d389e2ab91002584ac0be1dac28d87d5422137c603dc26a04637e2ec6035"} Oct 31 03:11:29.573121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.571710 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" event={"ID":"c1605d8d-256f-42f5-b3fc-448e3dde1950","Type":"ContainerStarted","Data":"6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188"} Oct 31 03:11:29.573121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.571775 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" event={"ID":"c1605d8d-256f-42f5-b3fc-448e3dde1950","Type":"ContainerStarted","Data":"aa5aaf65280fcb2b4e1143267b30b11c020fca9a6dfc7956c30ae0a57c2eff16"} Oct 31 03:11:29.664243 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.664178 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podStartSLOduration=1.664157817 podStartE2EDuration="1.664157817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:29.627608838 +0000 UTC m=+971.308134965" watchObservedRunningTime="2026-10-31 03:11:29.664157817 +0000 UTC m=+971.344683945" Oct 31 03:11:29.846203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.835490 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:29.846203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:29.845988 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:30.820299 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:30.820241 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:30.820956 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:30.820923 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:30.835746 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:30.835711 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:30.836026 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:30.835996 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:31.450171 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.450122 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.676268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.676215 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.728294 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.728183 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.737650 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.737608 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:11:31.860770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.808189 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 31 03:11:31.860770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.816944 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:31.860770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.817028 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:31.860770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.817516 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.860770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.834522 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:31.860770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.834579 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:31.971065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:31.945550 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.299163 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:32.299119 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 03:11:32.484862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:32.467338 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Oct 31 03:11:32.484862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:32.468020 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.610117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:32.590796 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:11:32.704162 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:32.704108 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.722252 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:32.722211 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:11:32.791205 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:32.743462 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:11:32.791205 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:32.744320 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:11:32.970081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:32.941666 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podStartSLOduration=4.94164387 podStartE2EDuration="4.94164387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:29.702421074 +0000 UTC m=+971.382947201" watchObservedRunningTime="2026-10-31 03:11:32.94164387 +0000 UTC m=+974.622169998" Oct 31 03:11:33.057107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.023423 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.057107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.035273 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:11:33.083092 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:11:33.072725 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:11:33.153707 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.153669 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:33.153985 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.153955 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:33.154217 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.154197 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:33.214933 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.214887 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:33.215207 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.158815 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:11:33.372183 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.368292 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.489573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.489399 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.768403 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.768363 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:11:33.791190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.791141 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.833385 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.833349 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:33.845458 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.845415 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:33.846147 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.834191 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:33.864101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:33.846327 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:34.096115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.072158 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 03:11:34.171189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.171141 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.335085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.332237 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.432397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.431787 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:11:34.718106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.534516 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.718106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.683301 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.774109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.774057 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.809067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.789376 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:11:34.822500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.822460 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:34.823738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.823700 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:34.836070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.834425 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:34.836070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:34.834481 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:35.123120 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.097955 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 03:11:35.211061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.208121 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:11:35.211061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.209211 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:11:35.258173 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.258125 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.737060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.734161 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.737060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.734570 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.737060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.734828 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:11:35.737060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.735441 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:11:35.790063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.789306 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.857964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.857925 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:35.858265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.858233 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:35.858445 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.858430 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:35.858569 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:35.858551 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:36.147121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.144918 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.208649 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.208596 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.274341 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.274295 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.275352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.275328 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:11:36.708086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.686497 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.727064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.725192 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:36.727064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.725252 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.737174 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.737125 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.749064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.746151 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.749064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.746694 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.790747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.790702 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.912059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.911325 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:36.912059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.911397 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:36.912059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.911493 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:36.912059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:36.911521 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:37.146015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.145132 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.234603 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.234148 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.305059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.298178 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 03:11:37.734061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.733864 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.734061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.733932 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.748418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.744153 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.749428 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.748743 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.749428 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.749135 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.791376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.764960 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.791376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.765018 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.791376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.765135 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.791376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.765171 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.793187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.793148 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.829130 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.828963 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:37.829130 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.829029 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:37.849427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.849306 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:37.849427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.849367 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:37.859061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.852160 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.859061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:37.852210 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.102349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.102231 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:38.102645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.102596 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.151330 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.150162 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.236203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.236151 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.340187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.340124 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.754029 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.753987 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.754742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.754322 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.755169 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.754358 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 03:11:38.842063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.838588 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.985285 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.984431 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:38.985285 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:38.984494 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:39.069132 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.028366 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.069132 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.028430 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:39.150408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.150362 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.274113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.274070 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.353422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.353317 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.757754 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.757708 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.758410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.758374 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.759014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.757661 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.817555 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.817520 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.817860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.817815 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:39.836514 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.836480 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.836795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.836750 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:39.847190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.847144 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.935215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:39.935164 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" podUID="5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.40:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.166678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.159282 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 03:11:40.276182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.276121 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.356242 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.356182 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.589879 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.589792 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:11:40.687987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.682313 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:11:40.741948 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.741916 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:11:40.783696 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.766596 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:11:40.783696 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.766929 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:11:40.818967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.818524 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:40.818967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.818583 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:40.854990 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.854901 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:40.855258 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.855222 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:40.929410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.927778 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:11:40.929410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:40.928577 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:11:41.044402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:41.044367 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:11:41.232111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:41.223223 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:11:41.232111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:41.223296 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:11:41.819133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:41.819095 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:41.819696 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:41.819663 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:41.835928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:41.835805 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:41.835928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:41.835863 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:42.819865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:42.819803 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:42.820390 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:42.819890 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:42.834503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:42.834471 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:42.834768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:42.834742 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:43.817115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:43.816204 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:43.818079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:43.817417 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:43.818079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:43.817474 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:43.833747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:43.833718 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:43.834405 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:43.834369 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:43.834531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:43.834416 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:44.818246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:44.816178 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:44.818246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:44.816235 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:44.838411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:44.838348 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:44.838919 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:44.838416 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:11:45.904112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:45.900239 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:11:45.904112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:45.900322 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:11:46.840483 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:46.840440 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:46.840740 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:46.840712 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:47.877057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:47.874148 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:47.877057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:47.874215 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:47.907098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:47.898124 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:47.907098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:47.898189 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:48.410357 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:48.410318 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:48.412742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:48.412720 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:11:48.617325 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:48.617291 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:48.618572 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:48.618545 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:11:53.023075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:53.021128 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6971-c8d550"] Oct 31 03:11:53.047094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:53.047061 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-c8d550" Oct 31 03:11:53.054616 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:53.053129 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bshq7\"" Oct 31 03:11:53.057341 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:53.056438 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6971-c8d550"] Oct 31 03:11:53.187200 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:53.187168 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxk5l\" (UniqueName: \"kubernetes.io/projected/79ae1169-93d1-485d-a3a1-6e2a7c7f451e-kube-api-access-mxk5l\") pod \"test-curl-6971-c8d550\" (UID: \"79ae1169-93d1-485d-a3a1-6e2a7c7f451e\") " pod="opendatahub/test-curl-6971-c8d550" Oct 31 03:11:53.319226 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:53.308011 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxk5l\" (UniqueName: \"kubernetes.io/projected/79ae1169-93d1-485d-a3a1-6e2a7c7f451e-kube-api-access-mxk5l\") pod \"test-curl-6971-c8d550\" (UID: \"79ae1169-93d1-485d-a3a1-6e2a7c7f451e\") " pod="opendatahub/test-curl-6971-c8d550" Oct 31 03:11:53.325790 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:53.325755 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxk5l\" (UniqueName: \"kubernetes.io/projected/79ae1169-93d1-485d-a3a1-6e2a7c7f451e-kube-api-access-mxk5l\") pod \"test-curl-6971-c8d550\" (UID: \"79ae1169-93d1-485d-a3a1-6e2a7c7f451e\") " pod="opendatahub/test-curl-6971-c8d550" Oct 31 03:11:53.386402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:53.386362 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-c8d550" Oct 31 03:11:53.701435 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:53.701400 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6971-c8d550"] Oct 31 03:11:53.703335 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:53.703314 4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:11:54.622821 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:54.622783 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6971-c8d550" event={"ID":"79ae1169-93d1-485d-a3a1-6e2a7c7f451e","Type":"ContainerStarted","Data":"acd99974598fb5b73a6e3ba974d0f3ea294ae4ab33763f61be461265b30ca345"} Oct 31 03:11:57.656118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:57.656076 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6971-c8d550" event={"ID":"79ae1169-93d1-485d-a3a1-6e2a7c7f451e","Type":"ContainerStarted","Data":"af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62"} Oct 31 03:11:57.679312 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:57.679252 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6971-c8d550" podStartSLOduration=1.321673083 podStartE2EDuration="4.679238049s" totalImagesPullingTime="3.357564966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:57.675745019 +0000 UTC m=+999.356271159" watchObservedRunningTime="2026-10-31 03:11:57.679238049 +0000 UTC m=+999.359764197" Oct 31 03:11:58.560812 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.560777 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9"] Oct 31 03:11:58.566372 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.566339 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.585465 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.584012 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9"] Oct 31 03:11:58.604026 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.603984 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6971-c8d550"] Oct 31 03:11:58.629457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.629411 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6971-c8d550"] Oct 31 03:11:58.660797 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.660756 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhvx\" (UniqueName: \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-kube-api-access-plhvx\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.661313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.660870 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.661313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.660918 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.661313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.661007 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.661313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.661067 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b501950-8aba-4108-9320-d59694798c0a-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.661313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.661113 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.661313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.661293 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.661627 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.661322 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.661627 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.661346 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b501950-8aba-4108-9320-d59694798c0a-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.763101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.762957 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.763442 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.763418 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.763668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.763649 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.763810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.763794 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b501950-8aba-4108-9320-d59694798c0a-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.763973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.763957 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.764290 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.764270 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.765694 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.765662 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.765854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.765823 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b501950-8aba-4108-9320-d59694798c0a-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.766005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.765986 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhvx\" (UniqueName: \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-kube-api-access-plhvx\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.766281 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.764929 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.766406 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.766003 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.766511 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.765288 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.766739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.766713 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b501950-8aba-4108-9320-d59694798c0a-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.766825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.765219 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.768076 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.767980 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.770169 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.768676 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b501950-8aba-4108-9320-d59694798c0a-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.782672 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.781912 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.783695 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.783645 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhvx\" (UniqueName: \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-kube-api-access-plhvx\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:58.892369 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:58.892269 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:11:59.191377 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:11:59.191333 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b501950_8aba_4108_9320_d59694798c0a.slice/crio-5fa2e057b2a06589b2615e5d8635e02e0b46b9598783486c03b16ef837c8c53c WatchSource:0}: Error finding container 5fa2e057b2a06589b2615e5d8635e02e0b46b9598783486c03b16ef837c8c53c: Status 404 returned error can't find the container with id 5fa2e057b2a06589b2615e5d8635e02e0b46b9598783486c03b16ef837c8c53c Oct 31 03:11:59.194229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:59.194202 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9"] Oct 31 03:11:59.194598 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:59.194564 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:11:59.194654 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:59.194628 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:11:59.194692 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:59.194658 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:11:59.679370 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:59.679336 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" event={"ID":"7b501950-8aba-4108-9320-d59694798c0a","Type":"ContainerStarted","Data":"b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1"} Oct 31 03:11:59.679900 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:59.679399 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" event={"ID":"7b501950-8aba-4108-9320-d59694798c0a","Type":"ContainerStarted","Data":"5fa2e057b2a06589b2615e5d8635e02e0b46b9598783486c03b16ef837c8c53c"} Oct 31 03:11:59.679900 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:59.679527 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6971-c8d550" podUID="79ae1169-93d1-485d-a3a1-6e2a7c7f451e" containerName="test-curl-6971-c8d550" containerID="cri-o://af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62" gracePeriod=2 Oct 31 03:11:59.685082 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:11:59.683303 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6971-c8d550\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="79ae1169-93d1-485d-a3a1-6e2a7c7f451e" pod="opendatahub/test-curl-6971-c8d550" Oct 31 03:11:59.726771 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:59.726719 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" podStartSLOduration=1.726706172 podStartE2EDuration="1.726706172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:59.724442852 +0000 UTC m=+1001.404968980" watchObservedRunningTime="2026-10-31 03:11:59.726706172 +0000 UTC m=+1001.407232298" Oct 31 03:11:59.727676 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:11:59.727646 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6971-c8d550\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="79ae1169-93d1-485d-a3a1-6e2a7c7f451e" pod="opendatahub/test-curl-6971-c8d550" Oct 31 03:11:59.895643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:11:59.895607 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:12:00.894416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:00.894384 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:12:00.894841 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:00.894444 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:12:01.893384 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:01.893020 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:12:01.893384 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:01.893099 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:12:02.094404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.094370 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-c8d550" Oct 31 03:12:02.137090 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:12:02.137022 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:12:02.145744 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.145714 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ae1169-93d1-485d-a3a1-6e2a7c7f451e-kube-api-access-mxk5l\" (UniqueName: \"kubernetes.io/projected/79ae1169-93d1-485d-a3a1-6e2a7c7f451e-kube-api-access-mxk5l\") pod \"79ae1169-93d1-485d-a3a1-6e2a7c7f451e\" (UID: \"79ae1169-93d1-485d-a3a1-6e2a7c7f451e\") " Oct 31 03:12:02.189291 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.186242 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ae1169-93d1-485d-a3a1-6e2a7c7f451e-kube-api-access-mxk5l" pod "79ae1169-93d1-485d-a3a1-6e2a7c7f451e" (UID: "79ae1169-93d1-485d-a3a1-6e2a7c7f451e"). InnerVolumeSpecName "kube-api-access-mxk5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:02.264188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.251316 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxk5l\" (UniqueName: \"kubernetes.io/projected/79ae1169-93d1-485d-a3a1-6e2a7c7f451e-kube-api-access-mxk5l\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:02.436744 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.436712 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ae1169-93d1-485d-a3a1-6e2a7c7f451e" path="/var/lib/kubelet/pods/79ae1169-93d1-485d-a3a1-6e2a7c7f451e/volumes" Oct 31 03:12:02.708408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.708322 4456 generic.go:356] "Generic (PLEG): container finished" podID="79ae1169-93d1-485d-a3a1-6e2a7c7f451e" containerID="af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62" exitCode=137 Oct 31 03:12:02.708408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.708362 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-c8d550" Oct 31 03:12:02.708658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.708416 4456 scope.go:122] "RemoveContainer" containerID="af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62" Oct 31 03:12:02.718554 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.718533 4456 scope.go:122] "RemoveContainer" containerID="af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62" Oct 31 03:12:02.718878 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:12:02.718851 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62\": container with ID starting with af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62 not found: ID does not exist" containerID="af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62" Oct 31 03:12:02.718943 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.718887 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62"} err="failed to get container status \"af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62\": rpc error: code = NotFound desc = could not find container \"af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62\": container with ID starting with af19a8c4eec6533934434a614232c9bd354edd9752c3996b377f864546545c62 not found: ID does not exist" Oct 31 03:12:02.892981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.892943 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:12:02.893164 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:02.893012 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:12:03.893088 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:03.893052 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:12:03.893584 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:03.893114 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:12:04.903059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:04.898472 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:12:04.903059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:04.898533 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:12:05.893597 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:05.893556 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:12:05.893784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:05.893622 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:12:06.899066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:06.895460 4456 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:12:06.899066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:06.895527 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:12:07.898090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:07.898019 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:12:07.898543 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:07.898519 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:12:07.899366 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:07.899344 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:12:11.749059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:11.748957 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.761067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:11.751250 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.802446 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:11.802109 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 31 03:12:12.132538 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.132433 4456 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:12:12.205165 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.205113 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.205619 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.145123 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.246272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.246228 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:12:12.358077 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.357162 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.576928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.576878 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.763065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.762148 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.763065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.762599 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:12:12.763679 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.763371 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:12:12.783061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.765116 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.783061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.765430 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:12:12.783061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:12.776513 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:12:13.236060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.235220 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 03:12:13.236060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.235944 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:12:13.270073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.267433 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.270073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.268652 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:12:13.393177 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.384116 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.393177 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.384580 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.420060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.403183 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:12:13.779471 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.779432 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.780349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.780323 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:12:13.780779 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.780759 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:12:13.803717 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.803669 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.847131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.847095 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:12:13.847454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.847436 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:12:13.863522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:13.863404 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.237181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.237133 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.278185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.278127 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.371134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.369691 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.402098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.399873 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.799533 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.783122 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.799533 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.784442 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.812069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.804829 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.846247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.846197 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.865528 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.849765 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hk27q" podUID="5b013bce-d4ab-48d5-8e6e-fa2bb6d80aa7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.40:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.866511 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.866479 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94"] Oct 31 03:12:14.867672 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.867646 4456 cpu_manager.go:414] "containerMap: removing container" podUID="79ae1169-93d1-485d-a3a1-6e2a7c7f451e" containerName="test-curl-6971-c8d550" Oct 31 03:12:14.872117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.872093 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ae1169-93d1-485d-a3a1-6e2a7c7f451e" containerName="test-curl-6971-c8d550" Oct 31 03:12:14.876107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.872522 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ae1169-93d1-485d-a3a1-6e2a7c7f451e" containerName="test-curl-6971-c8d550" Oct 31 03:12:14.969918 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.969886 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:14.981253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.981226 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:12:14.983575 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:14.983554 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:12:15.047126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.047093 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94"] Oct 31 03:12:15.085763 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.073971 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24663fd5-3c33-4656-be24-0126fe845968-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:15.085763 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.083482 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwlq\" (UniqueName: \"kubernetes.io/projected/24663fd5-3c33-4656-be24-0126fe845968-kube-api-access-6gwlq\") pod \"maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:15.085763 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.083641 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24663fd5-3c33-4656-be24-0126fe845968-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:15.248950 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.248912 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24663fd5-3c33-4656-be24-0126fe845968-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:15.249265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.249232 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 03:12:15.302061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.185029 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24663fd5-3c33-4656-be24-0126fe845968-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:15.302061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.279281 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwlq\" (UniqueName: \"kubernetes.io/projected/24663fd5-3c33-4656-be24-0126fe845968-kube-api-access-6gwlq\") pod \"maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:15.302061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.279409 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24663fd5-3c33-4656-be24-0126fe845968-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:15.302061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.280400 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.338621 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.338511 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwlq\" (UniqueName: \"kubernetes.io/projected/24663fd5-3c33-4656-be24-0126fe845968-kube-api-access-6gwlq\") pod \"maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:15.344689 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.344657 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24663fd5-3c33-4656-be24-0126fe845968-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:15.404175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.404130 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.624381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.624291 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:15.671677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.671648 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:12:15.741981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.741951 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:12:15.806394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.806352 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.816501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.816461 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:12:15.849845 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.849801 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.852628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:15.852591 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:12:16.080610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:16.080361 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:12:16.250061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:16.249465 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:12:16.326053 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:16.326008 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:12:16.346528 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:16.346501 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:12:16.347628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:16.347604 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94"] Oct 31 03:12:17.155083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:17.155049 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" event={"ID":"24663fd5-3c33-4656-be24-0126fe845968","Type":"ContainerStarted","Data":"c4c8421100e8ebf558c6bad35be960f5153b671974ec9f2813e763ae25611b0f"} Oct 31 03:12:17.155481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:17.155101 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" event={"ID":"24663fd5-3c33-4656-be24-0126fe845968","Type":"ContainerStarted","Data":"81bc4db0c4cee9627c460ee5839f2f1ef6c059f9436f5004777849831b5db447"} Oct 31 03:12:17.182888 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:17.182830 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" podStartSLOduration=3.182816655 podStartE2EDuration="3.182816655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:17.177681289 +0000 UTC m=+1018.858207421" watchObservedRunningTime="2026-10-31 03:12:17.182816655 +0000 UTC m=+1018.863342835" Oct 31 03:12:18.174082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:18.166877 4456 generic.go:356] "Generic (PLEG): container finished" podID="24663fd5-3c33-4656-be24-0126fe845968" containerID="c4c8421100e8ebf558c6bad35be960f5153b671974ec9f2813e763ae25611b0f" exitCode=0 Oct 31 03:12:18.174082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:18.169136 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" event={"ID":"24663fd5-3c33-4656-be24-0126fe845968","Type":"ContainerDied","Data":"c4c8421100e8ebf558c6bad35be960f5153b671974ec9f2813e763ae25611b0f"} Oct 31 03:12:19.356336 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:19.356301 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:19.401419 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:19.401380 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24663fd5-3c33-4656-be24-0126fe845968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24663fd5-3c33-4656-be24-0126fe845968-maas-api-xks-ca\") pod \"24663fd5-3c33-4656-be24-0126fe845968\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " Oct 31 03:12:19.401609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:19.401515 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24663fd5-3c33-4656-be24-0126fe845968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24663fd5-3c33-4656-be24-0126fe845968-maas-api-openshift-ca\") pod \"24663fd5-3c33-4656-be24-0126fe845968\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " Oct 31 03:12:19.401685 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:19.401625 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24663fd5-3c33-4656-be24-0126fe845968-kube-api-access-6gwlq\" (UniqueName: \"kubernetes.io/projected/24663fd5-3c33-4656-be24-0126fe845968-kube-api-access-6gwlq\") pod \"24663fd5-3c33-4656-be24-0126fe845968\" (UID: \"24663fd5-3c33-4656-be24-0126fe845968\") " Oct 31 03:12:19.402349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:19.402317 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24663fd5-3c33-4656-be24-0126fe845968-maas-api-openshift-ca" pod "24663fd5-3c33-4656-be24-0126fe845968" (UID: "24663fd5-3c33-4656-be24-0126fe845968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:19.407176 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:19.407150 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24663fd5-3c33-4656-be24-0126fe845968-maas-api-xks-ca" pod "24663fd5-3c33-4656-be24-0126fe845968" (UID: "24663fd5-3c33-4656-be24-0126fe845968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:19.407271 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:19.407192 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24663fd5-3c33-4656-be24-0126fe845968-kube-api-access-6gwlq" pod "24663fd5-3c33-4656-be24-0126fe845968" (UID: "24663fd5-3c33-4656-be24-0126fe845968"). InnerVolumeSpecName "kube-api-access-6gwlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:19.503709 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:19.503635 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24663fd5-3c33-4656-be24-0126fe845968-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:19.503709 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:19.503662 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gwlq\" (UniqueName: \"kubernetes.io/projected/24663fd5-3c33-4656-be24-0126fe845968-kube-api-access-6gwlq\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:19.503709 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:19.503671 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24663fd5-3c33-4656-be24-0126fe845968-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:20.186685 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:20.186653 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" Oct 31 03:12:20.186848 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:20.186650 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94" event={"ID":"24663fd5-3c33-4656-be24-0126fe845968","Type":"ContainerDied","Data":"81bc4db0c4cee9627c460ee5839f2f1ef6c059f9436f5004777849831b5db447"} Oct 31 03:12:20.186848 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:20.186789 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81bc4db0c4cee9627c460ee5839f2f1ef6c059f9436f5004777849831b5db447" Oct 31 03:12:32.159378 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:12:32.159270 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:12:34.272345 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:34.272310 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94"] Oct 31 03:12:34.281120 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:34.281082 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ac00dc5-prq94"] Oct 31 03:12:34.445058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:34.442672 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24663fd5-3c33-4656-be24-0126fe845968" path="/var/lib/kubelet/pods/24663fd5-3c33-4656-be24-0126fe845968/volumes" Oct 31 03:12:37.220138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.220101 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46"] Oct 31 03:12:37.220841 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.220813 4456 cpu_manager.go:414] "containerMap: removing container" podUID="24663fd5-3c33-4656-be24-0126fe845968" containerName="revoke-keys" Oct 31 03:12:37.220841 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.220831 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24663fd5-3c33-4656-be24-0126fe845968" containerName="revoke-keys" Oct 31 03:12:37.221056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.220992 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="24663fd5-3c33-4656-be24-0126fe845968" containerName="revoke-keys" Oct 31 03:12:37.224639 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.224615 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:12:37.257668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.257639 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-537aca72\"" Oct 31 03:12:37.260978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.260943 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46"] Oct 31 03:12:37.268307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.268286 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8"] Oct 31 03:12:37.273445 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.273420 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:12:37.313900 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.313848 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-537aca72-dockercfg-wshfv\"" Oct 31 03:12:37.342579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.342126 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9774m\" (UniqueName: \"kubernetes.io/projected/9b70f51a-43ed-486b-8eff-97008cf47dbd-kube-api-access-9774m\") pod \"payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46\" (UID: \"9b70f51a-43ed-486b-8eff-97008cf47dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:12:37.342579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.342331 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b70f51a-43ed-486b-8eff-97008cf47dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46\" (UID: \"9b70f51a-43ed-486b-8eff-97008cf47dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:12:37.417939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.417903 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8"] Oct 31 03:12:37.462061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.448998 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b70f51a-43ed-486b-8eff-97008cf47dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46\" (UID: \"9b70f51a-43ed-486b-8eff-97008cf47dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:12:37.462061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.449333 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9774m\" (UniqueName: \"kubernetes.io/projected/9b70f51a-43ed-486b-8eff-97008cf47dbd-kube-api-access-9774m\") pod \"payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46\" (UID: \"9b70f51a-43ed-486b-8eff-97008cf47dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:12:37.462061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.449379 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2p9\" (UniqueName: \"kubernetes.io/projected/9cee4444-a35b-4644-aaf3-6ec369ffbced-kube-api-access-rl2p9\") pod \"payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8\" (UID: \"9cee4444-a35b-4644-aaf3-6ec369ffbced\") " pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:12:37.465589 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.465488 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cee4444-a35b-4644-aaf3-6ec369ffbced-plugins-config-volume\") pod \"payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8\" (UID: \"9cee4444-a35b-4644-aaf3-6ec369ffbced\") " pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:12:37.525118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.466534 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b70f51a-43ed-486b-8eff-97008cf47dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46\" (UID: \"9b70f51a-43ed-486b-8eff-97008cf47dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:12:37.569977 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.569942 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cee4444-a35b-4644-aaf3-6ec369ffbced-plugins-config-volume\") pod \"payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8\" (UID: \"9cee4444-a35b-4644-aaf3-6ec369ffbced\") " pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:12:37.572099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.572073 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2p9\" (UniqueName: \"kubernetes.io/projected/9cee4444-a35b-4644-aaf3-6ec369ffbced-kube-api-access-rl2p9\") pod \"payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8\" (UID: \"9cee4444-a35b-4644-aaf3-6ec369ffbced\") " pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:12:37.572615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.571734 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cee4444-a35b-4644-aaf3-6ec369ffbced-plugins-config-volume\") pod \"payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8\" (UID: \"9cee4444-a35b-4644-aaf3-6ec369ffbced\") " pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:12:37.587070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.587015 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9774m\" (UniqueName: \"kubernetes.io/projected/9b70f51a-43ed-486b-8eff-97008cf47dbd-kube-api-access-9774m\") pod \"payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46\" (UID: \"9b70f51a-43ed-486b-8eff-97008cf47dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:12:37.590392 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.590370 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2p9\" (UniqueName: \"kubernetes.io/projected/9cee4444-a35b-4644-aaf3-6ec369ffbced-kube-api-access-rl2p9\") pod \"payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8\" (UID: \"9cee4444-a35b-4644-aaf3-6ec369ffbced\") " pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:12:37.609869 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.609834 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:12:37.846706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.846432 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:12:37.964929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:37.964885 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8"] Oct 31 03:12:38.232472 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:12:38.232430 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b70f51a_43ed_486b_8eff_97008cf47dbd.slice/crio-31e51e50f9d47ea4242d64f5cbab5a05a8f559ecb1c6429e7c415e6d57319559 WatchSource:0}: Error finding container 31e51e50f9d47ea4242d64f5cbab5a05a8f559ecb1c6429e7c415e6d57319559: Status 404 returned error can't find the container with id 31e51e50f9d47ea4242d64f5cbab5a05a8f559ecb1c6429e7c415e6d57319559 Oct 31 03:12:38.233962 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:38.233936 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46"] Oct 31 03:12:38.594062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:38.593927 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" event={"ID":"9b70f51a-43ed-486b-8eff-97008cf47dbd","Type":"ContainerStarted","Data":"31e51e50f9d47ea4242d64f5cbab5a05a8f559ecb1c6429e7c415e6d57319559"} Oct 31 03:12:38.595768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:38.595737 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" event={"ID":"9cee4444-a35b-4644-aaf3-6ec369ffbced","Type":"ContainerStarted","Data":"e7debccffc8171b47696e3fb7409eea652472b0a84932ea8af270246d5c93210"} Oct 31 03:12:43.679200 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:43.679159 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" event={"ID":"9b70f51a-43ed-486b-8eff-97008cf47dbd","Type":"ContainerStarted","Data":"f605d8b7ed4ba9cf6d7cf015dc3a0345cf95008518ea5f67cd312169514c0477"} Oct 31 03:12:43.679715 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:43.679232 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:12:43.681122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:43.681092 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" event={"ID":"9cee4444-a35b-4644-aaf3-6ec369ffbced","Type":"ContainerStarted","Data":"fb5a2f5ce88be4a6b05427da1048872f646ceb8fc3871f075e44c89934a8ba8e"} Oct 31 03:12:43.681256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:43.681188 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:12:43.703887 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:43.703832 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" podStartSLOduration=2.012637968 podStartE2EDuration="6.703819253s" totalImagesPullingTime="4.691181285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:43.700528387 +0000 UTC m=+1045.381054516" watchObservedRunningTime="2026-10-31 03:12:43.703819253 +0000 UTC m=+1045.384345443" Oct 31 03:12:43.724118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:43.724073 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" podStartSLOduration=1.7908921389999999 podStartE2EDuration="6.724058841s" totalImagesPullingTime="4.933166702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:43.721320277 +0000 UTC m=+1045.401846427" watchObservedRunningTime="2026-10-31 03:12:43.724058841 +0000 UTC m=+1045.404584968" Oct 31 03:12:51.758063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:51.757025 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:12:51.794061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:51.791160 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.794061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:51.791476 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 03:12:51.831113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:51.812223 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.831113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:51.812944 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.850861 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:51.850817 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.263061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:52.258142 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.291060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:52.290126 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.415059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:52.413338 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.453234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:52.453146 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.825084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:53.824728 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:12:54.352562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:54.351679 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:54.422286 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:54.421616 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 03:12:54.719185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:54.719157 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:12:54.724961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:12:54.723398 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:13:02.166109 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:13:02.166062 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:13:12.383876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:12.383837 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8"] Oct 31 03:13:12.391576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:12.391543 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46"] Oct 31 03:13:17.414073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.413330 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" podUID="9b70f51a-43ed-486b-8eff-97008cf47dbd" containerName="payload-pre-processing" containerID="cri-o://f605d8b7ed4ba9cf6d7cf015dc3a0345cf95008518ea5f67cd312169514c0477" gracePeriod=25 Oct 31 03:13:17.416133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.416089 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" podUID="9cee4444-a35b-4644-aaf3-6ec369ffbced" containerName="payload-processing" containerID="cri-o://fb5a2f5ce88be4a6b05427da1048872f646ceb8fc3871f075e44c89934a8ba8e" gracePeriod=25 Oct 31 03:13:17.840501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.840464 4456 generic.go:356] "Generic (PLEG): container finished" podID="9b70f51a-43ed-486b-8eff-97008cf47dbd" containerID="f605d8b7ed4ba9cf6d7cf015dc3a0345cf95008518ea5f67cd312169514c0477" exitCode=0 Oct 31 03:13:17.840869 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.840727 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" event={"ID":"9b70f51a-43ed-486b-8eff-97008cf47dbd","Type":"ContainerDied","Data":"f605d8b7ed4ba9cf6d7cf015dc3a0345cf95008518ea5f67cd312169514c0477"} Oct 31 03:13:17.866961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.866635 4456 generic.go:356] "Generic (PLEG): container finished" podID="9cee4444-a35b-4644-aaf3-6ec369ffbced" containerID="fb5a2f5ce88be4a6b05427da1048872f646ceb8fc3871f075e44c89934a8ba8e" exitCode=0 Oct 31 03:13:17.866961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.866832 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" event={"ID":"9cee4444-a35b-4644-aaf3-6ec369ffbced","Type":"ContainerDied","Data":"fb5a2f5ce88be4a6b05427da1048872f646ceb8fc3871f075e44c89934a8ba8e"} Oct 31 03:13:17.881426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.881394 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:13:17.929305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.926694 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:13:17.929305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.927019 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b70f51a-43ed-486b-8eff-97008cf47dbd-kube-api-access-9774m\" (UniqueName: \"kubernetes.io/projected/9b70f51a-43ed-486b-8eff-97008cf47dbd-kube-api-access-9774m\") pod \"9b70f51a-43ed-486b-8eff-97008cf47dbd\" (UID: \"9b70f51a-43ed-486b-8eff-97008cf47dbd\") " Oct 31 03:13:17.929305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.927978 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b70f51a-43ed-486b-8eff-97008cf47dbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b70f51a-43ed-486b-8eff-97008cf47dbd-plugins-config-volume\") pod \"9b70f51a-43ed-486b-8eff-97008cf47dbd\" (UID: \"9b70f51a-43ed-486b-8eff-97008cf47dbd\") " Oct 31 03:13:17.929814 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.929446 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b70f51a-43ed-486b-8eff-97008cf47dbd-plugins-config-volume" pod "9b70f51a-43ed-486b-8eff-97008cf47dbd" (UID: "9b70f51a-43ed-486b-8eff-97008cf47dbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:17.932430 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:17.932400 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b70f51a-43ed-486b-8eff-97008cf47dbd-kube-api-access-9774m" pod "9b70f51a-43ed-486b-8eff-97008cf47dbd" (UID: "9b70f51a-43ed-486b-8eff-97008cf47dbd"). InnerVolumeSpecName "kube-api-access-9774m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:18.030066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.029963 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cee4444-a35b-4644-aaf3-6ec369ffbced-kube-api-access-rl2p9\" (UniqueName: \"kubernetes.io/projected/9cee4444-a35b-4644-aaf3-6ec369ffbced-kube-api-access-rl2p9\") pod \"9cee4444-a35b-4644-aaf3-6ec369ffbced\" (UID: \"9cee4444-a35b-4644-aaf3-6ec369ffbced\") " Oct 31 03:13:18.030066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.030050 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cee4444-a35b-4644-aaf3-6ec369ffbced-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cee4444-a35b-4644-aaf3-6ec369ffbced-plugins-config-volume\") pod \"9cee4444-a35b-4644-aaf3-6ec369ffbced\" (UID: \"9cee4444-a35b-4644-aaf3-6ec369ffbced\") " Oct 31 03:13:18.030446 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.030419 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cee4444-a35b-4644-aaf3-6ec369ffbced-plugins-config-volume" pod "9cee4444-a35b-4644-aaf3-6ec369ffbced" (UID: "9cee4444-a35b-4644-aaf3-6ec369ffbced"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:18.030800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.030743 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9774m\" (UniqueName: \"kubernetes.io/projected/9b70f51a-43ed-486b-8eff-97008cf47dbd-kube-api-access-9774m\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:18.030800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.030771 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b70f51a-43ed-486b-8eff-97008cf47dbd-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:18.030800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.030784 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cee4444-a35b-4644-aaf3-6ec369ffbced-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:18.032810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.032787 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cee4444-a35b-4644-aaf3-6ec369ffbced-kube-api-access-rl2p9" pod "9cee4444-a35b-4644-aaf3-6ec369ffbced" (UID: "9cee4444-a35b-4644-aaf3-6ec369ffbced"). InnerVolumeSpecName "kube-api-access-rl2p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:18.132396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.132357 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2p9\" (UniqueName: \"kubernetes.io/projected/9cee4444-a35b-4644-aaf3-6ec369ffbced-kube-api-access-rl2p9\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:18.879321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.878921 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" Oct 31 03:13:18.879321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.878913 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46" event={"ID":"9b70f51a-43ed-486b-8eff-97008cf47dbd","Type":"ContainerDied","Data":"31e51e50f9d47ea4242d64f5cbab5a05a8f559ecb1c6429e7c415e6d57319559"} Oct 31 03:13:18.879321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.879118 4456 scope.go:122] "RemoveContainer" containerID="f605d8b7ed4ba9cf6d7cf015dc3a0345cf95008518ea5f67cd312169514c0477" Oct 31 03:13:18.881832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.881807 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" event={"ID":"9cee4444-a35b-4644-aaf3-6ec369ffbced","Type":"ContainerDied","Data":"e7debccffc8171b47696e3fb7409eea652472b0a84932ea8af270246d5c93210"} Oct 31 03:13:18.881959 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.881920 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8" Oct 31 03:13:18.893166 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.893148 4456 scope.go:122] "RemoveContainer" containerID="fb5a2f5ce88be4a6b05427da1048872f646ceb8fc3871f075e44c89934a8ba8e" Oct 31 03:13:18.904975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.904931 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8"] Oct 31 03:13:18.911817 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.911796 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-537aca72-766dd7d995-t52b8"] Oct 31 03:13:18.925850 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.925825 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46"] Oct 31 03:13:18.930636 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:18.930611 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-537aca72-7f9b4f74c5-shx46"] Oct 31 03:13:20.204740 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.204709 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79"] Oct 31 03:13:20.205378 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.205362 4456 cpu_manager.go:414] "containerMap: removing container" podUID="9cee4444-a35b-4644-aaf3-6ec369ffbced" containerName="payload-processing" Oct 31 03:13:20.205426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.205379 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cee4444-a35b-4644-aaf3-6ec369ffbced" containerName="payload-processing" Oct 31 03:13:20.205426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.205408 4456 cpu_manager.go:414] "containerMap: removing container" podUID="9b70f51a-43ed-486b-8eff-97008cf47dbd" containerName="payload-pre-processing" Oct 31 03:13:20.205426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.205413 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b70f51a-43ed-486b-8eff-97008cf47dbd" containerName="payload-pre-processing" Oct 31 03:13:20.205534 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.205524 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cee4444-a35b-4644-aaf3-6ec369ffbced" containerName="payload-processing" Oct 31 03:13:20.205571 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.205539 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b70f51a-43ed-486b-8eff-97008cf47dbd" containerName="payload-pre-processing" Oct 31 03:13:20.210446 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.210419 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:13:20.216092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.216027 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj"] Oct 31 03:13:20.221349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.221269 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-7692c4\"" Oct 31 03:13:20.221349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.221335 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-7692c4-dockercfg-r4bkn\"" Oct 31 03:13:20.221741 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.221721 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79"] Oct 31 03:13:20.221845 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.221831 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:13:20.256400 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.254282 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj"] Oct 31 03:13:20.268805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.268769 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzcgd\" (UniqueName: \"kubernetes.io/projected/a2f9db23-301e-4897-848e-6e1a1582b5d3-kube-api-access-pzcgd\") pod \"payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj\" (UID: \"a2f9db23-301e-4897-848e-6e1a1582b5d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:13:20.269135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.269115 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkl6\" (UniqueName: \"kubernetes.io/projected/b9f44798-81be-45f0-ad96-c56832446ef6-kube-api-access-tbkl6\") pod \"payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79\" (UID: \"b9f44798-81be-45f0-ad96-c56832446ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:13:20.269279 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.269262 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f44798-81be-45f0-ad96-c56832446ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79\" (UID: \"b9f44798-81be-45f0-ad96-c56832446ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:13:20.269601 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.269583 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f9db23-301e-4897-848e-6e1a1582b5d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj\" (UID: \"a2f9db23-301e-4897-848e-6e1a1582b5d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:13:20.371751 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.371713 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f9db23-301e-4897-848e-6e1a1582b5d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj\" (UID: \"a2f9db23-301e-4897-848e-6e1a1582b5d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:13:20.371985 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.371966 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzcgd\" (UniqueName: \"kubernetes.io/projected/a2f9db23-301e-4897-848e-6e1a1582b5d3-kube-api-access-pzcgd\") pod \"payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj\" (UID: \"a2f9db23-301e-4897-848e-6e1a1582b5d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:13:20.372093 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.372074 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkl6\" (UniqueName: \"kubernetes.io/projected/b9f44798-81be-45f0-ad96-c56832446ef6-kube-api-access-tbkl6\") pod \"payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79\" (UID: \"b9f44798-81be-45f0-ad96-c56832446ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:13:20.372137 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.372120 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f44798-81be-45f0-ad96-c56832446ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79\" (UID: \"b9f44798-81be-45f0-ad96-c56832446ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:13:20.372714 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.372690 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f9db23-301e-4897-848e-6e1a1582b5d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj\" (UID: \"a2f9db23-301e-4897-848e-6e1a1582b5d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:13:20.372996 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.372974 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f44798-81be-45f0-ad96-c56832446ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79\" (UID: \"b9f44798-81be-45f0-ad96-c56832446ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:13:20.388805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.388115 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkl6\" (UniqueName: \"kubernetes.io/projected/b9f44798-81be-45f0-ad96-c56832446ef6-kube-api-access-tbkl6\") pod \"payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79\" (UID: \"b9f44798-81be-45f0-ad96-c56832446ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:13:20.438538 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.408461 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzcgd\" (UniqueName: \"kubernetes.io/projected/a2f9db23-301e-4897-848e-6e1a1582b5d3-kube-api-access-pzcgd\") pod \"payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj\" (UID: \"a2f9db23-301e-4897-848e-6e1a1582b5d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:13:20.494112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.489492 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b70f51a-43ed-486b-8eff-97008cf47dbd" path="/var/lib/kubelet/pods/9b70f51a-43ed-486b-8eff-97008cf47dbd/volumes" Oct 31 03:13:20.494112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.490182 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cee4444-a35b-4644-aaf3-6ec369ffbced" path="/var/lib/kubelet/pods/9cee4444-a35b-4644-aaf3-6ec369ffbced/volumes" Oct 31 03:13:20.566104 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.559683 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:13:20.568443 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.568397 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:13:20.897330 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.897276 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79"] Oct 31 03:13:20.897502 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:13:20.897389 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f44798_81be_45f0_ad96_c56832446ef6.slice/crio-f698765b073ea1f59f72d882196ce4d541dae342adb9090b529b67d50922ba51 WatchSource:0}: Error finding container f698765b073ea1f59f72d882196ce4d541dae342adb9090b529b67d50922ba51: Status 404 returned error can't find the container with id f698765b073ea1f59f72d882196ce4d541dae342adb9090b529b67d50922ba51 Oct 31 03:13:20.902766 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:20.902742 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj"] Oct 31 03:13:20.911180 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:13:20.911153 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f9db23_301e_4897_848e_6e1a1582b5d3.slice/crio-bf90f012a01091f8b6e236d43f478af240852948f937a9426c245b83a6c8d87c WatchSource:0}: Error finding container bf90f012a01091f8b6e236d43f478af240852948f937a9426c245b83a6c8d87c: Status 404 returned error can't find the container with id bf90f012a01091f8b6e236d43f478af240852948f937a9426c245b83a6c8d87c Oct 31 03:13:21.912946 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:21.912893 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" event={"ID":"b9f44798-81be-45f0-ad96-c56832446ef6","Type":"ContainerStarted","Data":"5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847"} Oct 31 03:13:21.913459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:21.913388 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" event={"ID":"b9f44798-81be-45f0-ad96-c56832446ef6","Type":"ContainerStarted","Data":"f698765b073ea1f59f72d882196ce4d541dae342adb9090b529b67d50922ba51"} Oct 31 03:13:21.913459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:21.913439 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:13:21.915169 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:21.915134 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" event={"ID":"a2f9db23-301e-4897-848e-6e1a1582b5d3","Type":"ContainerStarted","Data":"77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74"} Oct 31 03:13:21.915295 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:21.915174 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" event={"ID":"a2f9db23-301e-4897-848e-6e1a1582b5d3","Type":"ContainerStarted","Data":"bf90f012a01091f8b6e236d43f478af240852948f937a9426c245b83a6c8d87c"} Oct 31 03:13:21.915295 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:21.915193 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:13:21.940496 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:21.940437 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" podStartSLOduration=1.9404184820000001 podStartE2EDuration="1.940418482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:21.935787419 +0000 UTC m=+1083.616313549" watchObservedRunningTime="2026-10-31 03:13:21.940418482 +0000 UTC m=+1083.620944608" Oct 31 03:13:21.961172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:21.961110 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" podStartSLOduration=1.961096028 podStartE2EDuration="1.961096028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:21.957564149 +0000 UTC m=+1083.638090275" watchObservedRunningTime="2026-10-31 03:13:21.961096028 +0000 UTC m=+1083.641622155" Oct 31 03:13:32.136995 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:13:32.136952 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:13:32.926485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:32.926453 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:13:32.926652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:32.926519 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:13:33.931440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:33.931403 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v"] Oct 31 03:13:33.935892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:33.935870 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:13:33.936129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:33.936102 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw"] Oct 31 03:13:33.942871 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:33.942834 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:13:33.944078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:33.944020 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v"] Oct 31 03:13:33.950711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:33.950689 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw"] Oct 31 03:13:33.998437 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:33.998398 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-fd2e0b\"" Oct 31 03:13:34.032960 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.032925 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc484\" (UniqueName: \"kubernetes.io/projected/c02eb473-9cd5-4595-926d-e17429e58a8c-kube-api-access-pc484\") pod \"payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v\" (UID: \"c02eb473-9cd5-4595-926d-e17429e58a8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:13:34.033188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.033122 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02eb473-9cd5-4595-926d-e17429e58a8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v\" (UID: \"c02eb473-9cd5-4595-926d-e17429e58a8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:13:34.033188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.033162 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw\" (UID: \"e0d91df2-e63c-4fd5-ad16-f45a30251fbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:13:34.033337 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.033313 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6h9v\" (UniqueName: \"kubernetes.io/projected/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-kube-api-access-b6h9v\") pod \"payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw\" (UID: \"e0d91df2-e63c-4fd5-ad16-f45a30251fbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:13:34.151065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.150163 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02eb473-9cd5-4595-926d-e17429e58a8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v\" (UID: \"c02eb473-9cd5-4595-926d-e17429e58a8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:13:34.151065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.150233 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw\" (UID: \"e0d91df2-e63c-4fd5-ad16-f45a30251fbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:13:34.151065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.150347 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6h9v\" (UniqueName: \"kubernetes.io/projected/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-kube-api-access-b6h9v\") pod \"payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw\" (UID: \"e0d91df2-e63c-4fd5-ad16-f45a30251fbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:13:34.151065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.150499 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc484\" (UniqueName: \"kubernetes.io/projected/c02eb473-9cd5-4595-926d-e17429e58a8c-kube-api-access-pc484\") pod \"payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v\" (UID: \"c02eb473-9cd5-4595-926d-e17429e58a8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:13:34.153523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.151784 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02eb473-9cd5-4595-926d-e17429e58a8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v\" (UID: \"c02eb473-9cd5-4595-926d-e17429e58a8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:13:34.157059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.154450 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw\" (UID: \"e0d91df2-e63c-4fd5-ad16-f45a30251fbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:13:34.186152 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.178081 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc484\" (UniqueName: \"kubernetes.io/projected/c02eb473-9cd5-4595-926d-e17429e58a8c-kube-api-access-pc484\") pod \"payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v\" (UID: \"c02eb473-9cd5-4595-926d-e17429e58a8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:13:34.186152 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.178124 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6h9v\" (UniqueName: \"kubernetes.io/projected/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-kube-api-access-b6h9v\") pod \"payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw\" (UID: \"e0d91df2-e63c-4fd5-ad16-f45a30251fbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:13:34.275082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.273153 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:13:34.282091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.281821 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:13:34.596934 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:13:34.596902 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02eb473_9cd5_4595_926d_e17429e58a8c.slice/crio-ad3f26d04688d3e36f1e2608c591a32f31b6900404f3614ed245ad51705e6f68 WatchSource:0}: Error finding container ad3f26d04688d3e36f1e2608c591a32f31b6900404f3614ed245ad51705e6f68: Status 404 returned error can't find the container with id ad3f26d04688d3e36f1e2608c591a32f31b6900404f3614ed245ad51705e6f68 Oct 31 03:13:34.599438 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:13:34.599099 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d91df2_e63c_4fd5_ad16_f45a30251fbb.slice/crio-fce8f177f8e801f5e499808a5eed3648c1b7ecf1003018385c94d6c7622d28e3 WatchSource:0}: Error finding container fce8f177f8e801f5e499808a5eed3648c1b7ecf1003018385c94d6c7622d28e3: Status 404 returned error can't find the container with id fce8f177f8e801f5e499808a5eed3648c1b7ecf1003018385c94d6c7622d28e3 Oct 31 03:13:34.599438 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.599209 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v"] Oct 31 03:13:34.605964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:34.605933 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw"] Oct 31 03:13:35.180820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:35.180777 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" event={"ID":"e0d91df2-e63c-4fd5-ad16-f45a30251fbb","Type":"ContainerStarted","Data":"75411ede2639c94b6b132b65904fe4513c8f0f82ee2b59c1fc9e37ad98c24fa6"} Oct 31 03:13:35.181572 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:35.181491 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" event={"ID":"e0d91df2-e63c-4fd5-ad16-f45a30251fbb","Type":"ContainerStarted","Data":"fce8f177f8e801f5e499808a5eed3648c1b7ecf1003018385c94d6c7622d28e3"} Oct 31 03:13:35.181769 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:35.181756 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:13:35.183508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:35.183475 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" event={"ID":"c02eb473-9cd5-4595-926d-e17429e58a8c","Type":"ContainerStarted","Data":"7d99331efa3798c3f3dca169e1052a59eafef432e0967137c7af692ad0eeb29a"} Oct 31 03:13:35.183647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:35.183530 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" event={"ID":"c02eb473-9cd5-4595-926d-e17429e58a8c","Type":"ContainerStarted","Data":"ad3f26d04688d3e36f1e2608c591a32f31b6900404f3614ed245ad51705e6f68"} Oct 31 03:13:35.183647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:35.183578 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:13:35.205923 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:35.205851 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" podStartSLOduration=2.205838407 podStartE2EDuration="2.205838407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:35.202385915 +0000 UTC m=+1096.882912041" watchObservedRunningTime="2026-10-31 03:13:35.205838407 +0000 UTC m=+1096.886364576" Oct 31 03:13:35.231727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:35.231673 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" podStartSLOduration=2.231658754 podStartE2EDuration="2.231658754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:35.227960392 +0000 UTC m=+1096.908486519" watchObservedRunningTime="2026-10-31 03:13:35.231658754 +0000 UTC m=+1096.912184881" Oct 31 03:13:41.052051 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:41.051996 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr"] Oct 31 03:13:41.058658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:41.058618 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522c9223-sjrmr"] Oct 31 03:13:42.467600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:42.466897 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92126027-f6de-44d0-bdc1-2c156ecb60de" path="/var/lib/kubelet/pods/92126027-f6de-44d0-bdc1-2c156ecb60de/volumes" Oct 31 03:13:46.200309 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:46.200275 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:13:47.206640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:47.206611 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:13:50.258495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.258454 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q"] Oct 31 03:13:50.263492 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.263467 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:13:50.270644 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.270622 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-3fc2d0-dockercfg-dxktv\"" Oct 31 03:13:50.270876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.270851 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3fc2d0\"" Oct 31 03:13:50.274897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.274869 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q"] Oct 31 03:13:50.298867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.298825 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q\" (UID: \"b0ad8b22-0439-463d-9d1e-518f50f7f7e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:13:50.299088 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.299025 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdcj4\" (UniqueName: \"kubernetes.io/projected/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-kube-api-access-pdcj4\") pod \"payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q\" (UID: \"b0ad8b22-0439-463d-9d1e-518f50f7f7e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:13:50.402458 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.400738 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q\" (UID: \"b0ad8b22-0439-463d-9d1e-518f50f7f7e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:13:50.402693 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.402300 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q\" (UID: \"b0ad8b22-0439-463d-9d1e-518f50f7f7e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:13:50.406627 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.404414 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdcj4\" (UniqueName: \"kubernetes.io/projected/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-kube-api-access-pdcj4\") pod \"payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q\" (UID: \"b0ad8b22-0439-463d-9d1e-518f50f7f7e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:13:50.476059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.464656 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdcj4\" (UniqueName: \"kubernetes.io/projected/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-kube-api-access-pdcj4\") pod \"payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q\" (UID: \"b0ad8b22-0439-463d-9d1e-518f50f7f7e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:13:50.602117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.601143 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:13:50.895611 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:13:50.895550 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ad8b22_0439_463d_9d1e_518f50f7f7e4.slice/crio-d57a483fca5a50fee307168d0fc55cbdad638e7f8373c1f3c47490e81dc924ee WatchSource:0}: Error finding container d57a483fca5a50fee307168d0fc55cbdad638e7f8373c1f3c47490e81dc924ee: Status 404 returned error can't find the container with id d57a483fca5a50fee307168d0fc55cbdad638e7f8373c1f3c47490e81dc924ee Oct 31 03:13:50.897170 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:50.897145 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q"] Oct 31 03:13:51.528751 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:51.528710 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" event={"ID":"b0ad8b22-0439-463d-9d1e-518f50f7f7e4","Type":"ContainerStarted","Data":"be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0"} Oct 31 03:13:51.529265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:51.528766 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" event={"ID":"b0ad8b22-0439-463d-9d1e-518f50f7f7e4","Type":"ContainerStarted","Data":"d57a483fca5a50fee307168d0fc55cbdad638e7f8373c1f3c47490e81dc924ee"} Oct 31 03:13:51.529265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:51.528789 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:13:51.557185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:51.557138 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" podStartSLOduration=1.557124768 podStartE2EDuration="1.557124768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:51.551621935 +0000 UTC m=+1113.232148064" watchObservedRunningTime="2026-10-31 03:13:51.557124768 +0000 UTC m=+1113.237650894" Oct 31 03:13:58.849768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:58.849163 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg"] Oct 31 03:13:58.856687 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:58.856657 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:13:58.857622 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:58.857594 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg"] Oct 31 03:13:58.866955 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:58.866930 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-677281\"" Oct 31 03:13:58.973107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:58.972911 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmkgk\" (UniqueName: \"kubernetes.io/projected/30e07052-2f47-420f-a107-c9b3c51b210d-kube-api-access-mmkgk\") pod \"payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg\" (UID: \"30e07052-2f47-420f-a107-c9b3c51b210d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:13:58.973482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:58.973460 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e07052-2f47-420f-a107-c9b3c51b210d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg\" (UID: \"30e07052-2f47-420f-a107-c9b3c51b210d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:13:59.085098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:59.084225 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmkgk\" (UniqueName: \"kubernetes.io/projected/30e07052-2f47-420f-a107-c9b3c51b210d-kube-api-access-mmkgk\") pod \"payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg\" (UID: \"30e07052-2f47-420f-a107-c9b3c51b210d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:13:59.085098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:59.084358 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e07052-2f47-420f-a107-c9b3c51b210d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg\" (UID: \"30e07052-2f47-420f-a107-c9b3c51b210d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:13:59.085406 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:59.085380 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e07052-2f47-420f-a107-c9b3c51b210d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg\" (UID: \"30e07052-2f47-420f-a107-c9b3c51b210d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:13:59.124385 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:59.124279 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmkgk\" (UniqueName: \"kubernetes.io/projected/30e07052-2f47-420f-a107-c9b3c51b210d-kube-api-access-mmkgk\") pod \"payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg\" (UID: \"30e07052-2f47-420f-a107-c9b3c51b210d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:13:59.185876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:59.185841 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:13:59.533676 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:59.533028 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg"] Oct 31 03:13:59.625238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:13:59.625180 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" event={"ID":"30e07052-2f47-420f-a107-c9b3c51b210d","Type":"ContainerStarted","Data":"a4a85c1a6fbd7c86056b85fc7fa5879e54766982b9ca25ed774b809908090094"} Oct 31 03:14:00.640523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:00.640480 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" event={"ID":"30e07052-2f47-420f-a107-c9b3c51b210d","Type":"ContainerStarted","Data":"bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce"} Oct 31 03:14:00.640971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:00.640553 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:14:00.665157 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:00.665098 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" podStartSLOduration=2.665075332 podStartE2EDuration="2.665075332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:00.661458419 +0000 UTC m=+1122.341984546" watchObservedRunningTime="2026-10-31 03:14:00.665075332 +0000 UTC m=+1122.345601459" Oct 31 03:14:02.139837 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:02.139796 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:14:02.540643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:02.540611 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:14:10.218571 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:10.218537 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-a0141a"] Oct 31 03:14:10.223302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:10.223275 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a0141a" Oct 31 03:14:10.227875 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:10.227849 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-a0141a"] Oct 31 03:14:10.230228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:10.230207 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bshq7\"" Oct 31 03:14:10.300014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:10.299980 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pprfl\" (UniqueName: \"kubernetes.io/projected/0603b9fb-fde2-41cf-8ffe-0b7edae49c3a-kube-api-access-pprfl\") pod \"test-curl-6968-a0141a\" (UID: \"0603b9fb-fde2-41cf-8ffe-0b7edae49c3a\") " pod="opendatahub/test-curl-6968-a0141a" Oct 31 03:14:10.401141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:10.401108 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pprfl\" (UniqueName: \"kubernetes.io/projected/0603b9fb-fde2-41cf-8ffe-0b7edae49c3a-kube-api-access-pprfl\") pod \"test-curl-6968-a0141a\" (UID: \"0603b9fb-fde2-41cf-8ffe-0b7edae49c3a\") " pod="opendatahub/test-curl-6968-a0141a" Oct 31 03:14:10.414775 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:10.414744 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pprfl\" (UniqueName: \"kubernetes.io/projected/0603b9fb-fde2-41cf-8ffe-0b7edae49c3a-kube-api-access-pprfl\") pod \"test-curl-6968-a0141a\" (UID: \"0603b9fb-fde2-41cf-8ffe-0b7edae49c3a\") " pod="opendatahub/test-curl-6968-a0141a" Oct 31 03:14:10.561447 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:10.561352 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a0141a" Oct 31 03:14:10.915080 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:14:10.915013 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0603b9fb_fde2_41cf_8ffe_0b7edae49c3a.slice/crio-5a564c47dd1d48c804b4086964181c513cda4d15c6da362ef70e85efd13f9e0f WatchSource:0}: Error finding container 5a564c47dd1d48c804b4086964181c513cda4d15c6da362ef70e85efd13f9e0f: Status 404 returned error can't find the container with id 5a564c47dd1d48c804b4086964181c513cda4d15c6da362ef70e85efd13f9e0f Oct 31 03:14:10.915931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:10.915854 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-a0141a"] Oct 31 03:14:11.653205 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:11.653167 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:14:11.760477 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:11.760441 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-a0141a" event={"ID":"0603b9fb-fde2-41cf-8ffe-0b7edae49c3a","Type":"ContainerStarted","Data":"c02febbf4b2b8c1b46752b0cd8d7b542d67fa08d956ad24f7d627b8803d46290"} Oct 31 03:14:11.760477 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:11.760483 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-a0141a" event={"ID":"0603b9fb-fde2-41cf-8ffe-0b7edae49c3a","Type":"ContainerStarted","Data":"5a564c47dd1d48c804b4086964181c513cda4d15c6da362ef70e85efd13f9e0f"} Oct 31 03:14:11.782363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:11.782311 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-a0141a" podStartSLOduration=1.78229571 podStartE2EDuration="1.78229571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:11.778748891 +0000 UTC m=+1133.459275020" watchObservedRunningTime="2026-10-31 03:14:11.78229571 +0000 UTC m=+1133.462821836" Oct 31 03:14:12.564713 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.564676 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-a0141a"] Oct 31 03:14:12.573019 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.572988 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-a0141a"] Oct 31 03:14:12.741091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.738971 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-06b47f"] Oct 31 03:14:12.741091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.740174 4456 cpu_manager.go:414] "containerMap: removing container" podUID="0603b9fb-fde2-41cf-8ffe-0b7edae49c3a" containerName="test-curl-6968-a0141a" Oct 31 03:14:12.741091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.740199 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0603b9fb-fde2-41cf-8ffe-0b7edae49c3a" containerName="test-curl-6968-a0141a" Oct 31 03:14:12.741091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.740369 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0603b9fb-fde2-41cf-8ffe-0b7edae49c3a" containerName="test-curl-6968-a0141a" Oct 31 03:14:12.746317 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.745843 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-06b47f" Oct 31 03:14:12.760081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.752083 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-06b47f"] Oct 31 03:14:12.868506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.868406 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwv7z\" (UniqueName: \"kubernetes.io/projected/2ba1122c-e0a4-4a82-9a93-9a8f55a069be-kube-api-access-gwv7z\") pod \"test-curl-6968-06b47f\" (UID: \"2ba1122c-e0a4-4a82-9a93-9a8f55a069be\") " pod="opendatahub/test-curl-6968-06b47f" Oct 31 03:14:12.969823 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.969770 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwv7z\" (UniqueName: \"kubernetes.io/projected/2ba1122c-e0a4-4a82-9a93-9a8f55a069be-kube-api-access-gwv7z\") pod \"test-curl-6968-06b47f\" (UID: \"2ba1122c-e0a4-4a82-9a93-9a8f55a069be\") " pod="opendatahub/test-curl-6968-06b47f" Oct 31 03:14:12.983133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:12.983100 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwv7z\" (UniqueName: \"kubernetes.io/projected/2ba1122c-e0a4-4a82-9a93-9a8f55a069be-kube-api-access-gwv7z\") pod \"test-curl-6968-06b47f\" (UID: \"2ba1122c-e0a4-4a82-9a93-9a8f55a069be\") " pod="opendatahub/test-curl-6968-06b47f" Oct 31 03:14:13.095621 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:13.095581 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-06b47f" Oct 31 03:14:13.381823 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:13.381797 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-06b47f"] Oct 31 03:14:13.809873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:13.808593 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-a0141a" podUID="0603b9fb-fde2-41cf-8ffe-0b7edae49c3a" containerName="test-curl-6968-a0141a" containerID="cri-o://c02febbf4b2b8c1b46752b0cd8d7b542d67fa08d956ad24f7d627b8803d46290" gracePeriod=2 Oct 31 03:14:13.809873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:13.809614 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-06b47f" event={"ID":"2ba1122c-e0a4-4a82-9a93-9a8f55a069be","Type":"ContainerStarted","Data":"b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca"} Oct 31 03:14:13.809873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:13.809653 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-06b47f" event={"ID":"2ba1122c-e0a4-4a82-9a93-9a8f55a069be","Type":"ContainerStarted","Data":"48e71ecf663938ed040e05742356bde4fd4b1a76e1590411bdfeed8c94829d51"} Oct 31 03:14:13.829195 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:13.829147 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-06b47f" podStartSLOduration=1.829134475 podStartE2EDuration="1.829134475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:13.828179174 +0000 UTC m=+1135.508705300" watchObservedRunningTime="2026-10-31 03:14:13.829134475 +0000 UTC m=+1135.509660601" Oct 31 03:14:13.831497 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:13.831459 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-a0141a\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="0603b9fb-fde2-41cf-8ffe-0b7edae49c3a" pod="opendatahub/test-curl-6968-a0141a" Oct 31 03:14:13.834752 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:13.834725 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-a0141a\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="0603b9fb-fde2-41cf-8ffe-0b7edae49c3a" pod="opendatahub/test-curl-6968-a0141a" Oct 31 03:14:14.634664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:14.634625 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-06b47f"] Oct 31 03:14:14.640606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:14.640577 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-06b47f"] Oct 31 03:14:14.812566 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:14.812531 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-a71084"] Oct 31 03:14:14.813455 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:14.813433 4456 cpu_manager.go:414] "containerMap: removing container" podUID="2ba1122c-e0a4-4a82-9a93-9a8f55a069be" containerName="test-curl-6968-06b47f" Oct 31 03:14:14.813579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:14.813455 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba1122c-e0a4-4a82-9a93-9a8f55a069be" containerName="test-curl-6968-06b47f" Oct 31 03:14:14.813639 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:14.813624 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba1122c-e0a4-4a82-9a93-9a8f55a069be" containerName="test-curl-6968-06b47f" Oct 31 03:14:14.837267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:14.837229 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-a71084"] Oct 31 03:14:14.837416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:14.837356 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:15.007181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:15.007137 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtk4\" (UniqueName: \"kubernetes.io/projected/bffa20b5-7f4f-4160-888d-31ae12bda586-kube-api-access-dgtk4\") pod \"test-curl-6968-a71084\" (UID: \"bffa20b5-7f4f-4160-888d-31ae12bda586\") " pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:15.108594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:15.108549 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtk4\" (UniqueName: \"kubernetes.io/projected/bffa20b5-7f4f-4160-888d-31ae12bda586-kube-api-access-dgtk4\") pod \"test-curl-6968-a71084\" (UID: \"bffa20b5-7f4f-4160-888d-31ae12bda586\") " pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:15.121683 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:15.121650 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtk4\" (UniqueName: \"kubernetes.io/projected/bffa20b5-7f4f-4160-888d-31ae12bda586-kube-api-access-dgtk4\") pod \"test-curl-6968-a71084\" (UID: \"bffa20b5-7f4f-4160-888d-31ae12bda586\") " pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:15.152329 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:15.152282 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:15.436355 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:14:15.436319 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbffa20b5_7f4f_4160_888d_31ae12bda586.slice/crio-bf18cea4d5859956372846e7a3bad9b4dc073746fdfd88b99c0dd5f13fba64a3 WatchSource:0}: Error finding container bf18cea4d5859956372846e7a3bad9b4dc073746fdfd88b99c0dd5f13fba64a3: Status 404 returned error can't find the container with id bf18cea4d5859956372846e7a3bad9b4dc073746fdfd88b99c0dd5f13fba64a3 Oct 31 03:14:15.437781 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:15.437755 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-a71084"] Oct 31 03:14:15.832374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:15.832327 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-a71084" event={"ID":"bffa20b5-7f4f-4160-888d-31ae12bda586","Type":"ContainerStarted","Data":"e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924"} Oct 31 03:14:15.832784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:15.832380 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-a71084" event={"ID":"bffa20b5-7f4f-4160-888d-31ae12bda586","Type":"ContainerStarted","Data":"bf18cea4d5859956372846e7a3bad9b4dc073746fdfd88b99c0dd5f13fba64a3"} Oct 31 03:14:15.834028 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:15.834002 4456 generic.go:356] "Generic (PLEG): container finished" podID="0603b9fb-fde2-41cf-8ffe-0b7edae49c3a" containerID="c02febbf4b2b8c1b46752b0cd8d7b542d67fa08d956ad24f7d627b8803d46290" exitCode=137 Oct 31 03:14:15.834231 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:15.834210 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-06b47f" podUID="2ba1122c-e0a4-4a82-9a93-9a8f55a069be" containerName="test-curl-6968-06b47f" containerID="cri-o://b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca" gracePeriod=2 Oct 31 03:14:15.867622 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:15.854949 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-06b47f\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="2ba1122c-e0a4-4a82-9a93-9a8f55a069be" pod="opendatahub/test-curl-6968-06b47f" Oct 31 03:14:15.867622 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:15.856825 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-a71084" podStartSLOduration=1.8568080820000001 podStartE2EDuration="1.856808082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:15.851350986 +0000 UTC m=+1137.531877113" watchObservedRunningTime="2026-10-31 03:14:15.856808082 +0000 UTC m=+1137.537334209" Oct 31 03:14:16.013974 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.013933 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a0141a" Oct 31 03:14:16.017618 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:16.017577 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-06b47f\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="2ba1122c-e0a4-4a82-9a93-9a8f55a069be" pod="opendatahub/test-curl-6968-06b47f" Oct 31 03:14:16.039508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.039473 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0603b9fb-fde2-41cf-8ffe-0b7edae49c3a-kube-api-access-pprfl\" (UniqueName: \"kubernetes.io/projected/0603b9fb-fde2-41cf-8ffe-0b7edae49c3a-kube-api-access-pprfl\") pod \"0603b9fb-fde2-41cf-8ffe-0b7edae49c3a\" (UID: \"0603b9fb-fde2-41cf-8ffe-0b7edae49c3a\") " Oct 31 03:14:16.041632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.041604 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0603b9fb-fde2-41cf-8ffe-0b7edae49c3a-kube-api-access-pprfl" pod "0603b9fb-fde2-41cf-8ffe-0b7edae49c3a" (UID: "0603b9fb-fde2-41cf-8ffe-0b7edae49c3a"). InnerVolumeSpecName "kube-api-access-pprfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:14:16.141455 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.141427 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pprfl\" (UniqueName: \"kubernetes.io/projected/0603b9fb-fde2-41cf-8ffe-0b7edae49c3a-kube-api-access-pprfl\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:14:16.437423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.437389 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0603b9fb-fde2-41cf-8ffe-0b7edae49c3a" path="/var/lib/kubelet/pods/0603b9fb-fde2-41cf-8ffe-0b7edae49c3a/volumes" Oct 31 03:14:16.805165 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.793332 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-a71084"] Oct 31 03:14:16.805165 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.798395 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-a71084"] Oct 31 03:14:16.859799 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.859766 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a0141a" Oct 31 03:14:16.860355 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.859817 4456 scope.go:122] "RemoveContainer" containerID="c02febbf4b2b8c1b46752b0cd8d7b542d67fa08d956ad24f7d627b8803d46290" Oct 31 03:14:16.880172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.875931 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6968-11e30e"] Oct 31 03:14:16.880172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.876980 4456 cpu_manager.go:414] "containerMap: removing container" podUID="bffa20b5-7f4f-4160-888d-31ae12bda586" containerName="test-curl-6968-a71084" Oct 31 03:14:16.880172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.877020 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffa20b5-7f4f-4160-888d-31ae12bda586" containerName="test-curl-6968-a71084" Oct 31 03:14:16.880172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.877256 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffa20b5-7f4f-4160-888d-31ae12bda586" containerName="test-curl-6968-a71084" Oct 31 03:14:16.884267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.884248 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-11e30e" Oct 31 03:14:16.897496 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.892158 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-11e30e"] Oct 31 03:14:16.960320 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:16.960282 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m676c\" (UniqueName: \"kubernetes.io/projected/8ac7217d-3a6d-4fd0-810c-26fe6762d0d5-kube-api-access-m676c\") pod \"test-curl-6968-11e30e\" (UID: \"8ac7217d-3a6d-4fd0-810c-26fe6762d0d5\") " pod="opendatahub/test-curl-6968-11e30e" Oct 31 03:14:17.061959 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:17.061855 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m676c\" (UniqueName: \"kubernetes.io/projected/8ac7217d-3a6d-4fd0-810c-26fe6762d0d5-kube-api-access-m676c\") pod \"test-curl-6968-11e30e\" (UID: \"8ac7217d-3a6d-4fd0-810c-26fe6762d0d5\") " pod="opendatahub/test-curl-6968-11e30e" Oct 31 03:14:17.080905 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:17.080875 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m676c\" (UniqueName: \"kubernetes.io/projected/8ac7217d-3a6d-4fd0-810c-26fe6762d0d5-kube-api-access-m676c\") pod \"test-curl-6968-11e30e\" (UID: \"8ac7217d-3a6d-4fd0-810c-26fe6762d0d5\") " pod="opendatahub/test-curl-6968-11e30e" Oct 31 03:14:17.208181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:17.207706 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-11e30e" Oct 31 03:14:17.697330 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:17.697299 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6968-11e30e"] Oct 31 03:14:17.703326 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:14:17.703295 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ac7217d_3a6d_4fd0_810c_26fe6762d0d5.slice/crio-26e214dd3b78264603a1b2720f8e17d2897465d4401e829cf2d416fd5346a61d WatchSource:0}: Error finding container 26e214dd3b78264603a1b2720f8e17d2897465d4401e829cf2d416fd5346a61d: Status 404 returned error can't find the container with id 26e214dd3b78264603a1b2720f8e17d2897465d4401e829cf2d416fd5346a61d Oct 31 03:14:17.933061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:17.923408 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-a71084" podUID="bffa20b5-7f4f-4160-888d-31ae12bda586" containerName="test-curl-6968-a71084" containerID="cri-o://e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924" gracePeriod=2 Oct 31 03:14:17.933061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:17.924541 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-11e30e" event={"ID":"8ac7217d-3a6d-4fd0-810c-26fe6762d0d5","Type":"ContainerStarted","Data":"e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3"} Oct 31 03:14:17.933061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:17.924584 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6968-11e30e" event={"ID":"8ac7217d-3a6d-4fd0-810c-26fe6762d0d5","Type":"ContainerStarted","Data":"26e214dd3b78264603a1b2720f8e17d2897465d4401e829cf2d416fd5346a61d"} Oct 31 03:14:17.933061 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:17.930210 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-a71084\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="bffa20b5-7f4f-4160-888d-31ae12bda586" pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:17.963505 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:17.963457 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-a71084\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="bffa20b5-7f4f-4160-888d-31ae12bda586" pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:17.972772 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:17.972692 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6968-11e30e" podStartSLOduration=1.972671995 podStartE2EDuration="1.972671995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:17.957581875 +0000 UTC m=+1139.638108023" watchObservedRunningTime="2026-10-31 03:14:17.972671995 +0000 UTC m=+1139.653198122" Oct 31 03:14:18.151934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.151899 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-06b47f" Oct 31 03:14:18.155453 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:18.155407 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-a71084\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="bffa20b5-7f4f-4160-888d-31ae12bda586" pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:18.210254 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.209437 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba1122c-e0a4-4a82-9a93-9a8f55a069be-kube-api-access-gwv7z\" (UniqueName: \"kubernetes.io/projected/2ba1122c-e0a4-4a82-9a93-9a8f55a069be-kube-api-access-gwv7z\") pod \"2ba1122c-e0a4-4a82-9a93-9a8f55a069be\" (UID: \"2ba1122c-e0a4-4a82-9a93-9a8f55a069be\") " Oct 31 03:14:18.216202 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.213693 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba1122c-e0a4-4a82-9a93-9a8f55a069be-kube-api-access-gwv7z" pod "2ba1122c-e0a4-4a82-9a93-9a8f55a069be" (UID: "2ba1122c-e0a4-4a82-9a93-9a8f55a069be"). InnerVolumeSpecName "kube-api-access-gwv7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:14:18.311439 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.311340 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwv7z\" (UniqueName: \"kubernetes.io/projected/2ba1122c-e0a4-4a82-9a93-9a8f55a069be-kube-api-access-gwv7z\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:14:18.444151 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.444116 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba1122c-e0a4-4a82-9a93-9a8f55a069be" path="/var/lib/kubelet/pods/2ba1122c-e0a4-4a82-9a93-9a8f55a069be/volumes" Oct 31 03:14:18.445956 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:18.445919 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-a71084\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="bffa20b5-7f4f-4160-888d-31ae12bda586" pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:18.802064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.799362 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6968-11e30e"] Oct 31 03:14:18.815775 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.815744 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6968-11e30e"] Oct 31 03:14:18.937940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.937895 4456 generic.go:356] "Generic (PLEG): container finished" podID="2ba1122c-e0a4-4a82-9a93-9a8f55a069be" containerID="b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca" exitCode=137 Oct 31 03:14:18.938471 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.937977 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-06b47f" Oct 31 03:14:18.938471 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.938030 4456 scope.go:122] "RemoveContainer" containerID="b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca" Oct 31 03:14:18.954724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.954699 4456 scope.go:122] "RemoveContainer" containerID="b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca" Oct 31 03:14:18.955082 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:18.955050 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca\": container with ID starting with b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca not found: ID does not exist" containerID="b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca" Oct 31 03:14:18.955150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:18.955091 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca"} err="failed to get container status \"b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca\": rpc error: code = NotFound desc = could not find container \"b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca\": container with ID starting with b981042acb32fefc6b41c970028b5b43b58faf2b58822b4a0ac24700180e36ca not found: ID does not exist" Oct 31 03:14:20.305122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:20.304609 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6968-11e30e" podUID="8ac7217d-3a6d-4fd0-810c-26fe6762d0d5" containerName="test-curl-6968-11e30e" containerID="cri-o://e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3" gracePeriod=2 Oct 31 03:14:20.352079 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:20.317462 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-11e30e\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="8ac7217d-3a6d-4fd0-810c-26fe6762d0d5" pod="opendatahub/test-curl-6968-11e30e" Oct 31 03:14:20.574710 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:20.574578 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7r8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:14:20.574993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:20.574937 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" podUID="ef33ec8e-50ff-4f0b-a84c-25a9b07c2221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.254060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:21.253457 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:21.315090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:21.313912 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffa20b5-7f4f-4160-888d-31ae12bda586-kube-api-access-dgtk4\" (UniqueName: \"kubernetes.io/projected/bffa20b5-7f4f-4160-888d-31ae12bda586-kube-api-access-dgtk4\") pod \"bffa20b5-7f4f-4160-888d-31ae12bda586\" (UID: \"bffa20b5-7f4f-4160-888d-31ae12bda586\") " Oct 31 03:14:21.337514 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:21.337460 4456 generic.go:356] "Generic (PLEG): container finished" podID="bffa20b5-7f4f-4160-888d-31ae12bda586" containerID="e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924" exitCode=137 Oct 31 03:14:21.338160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:21.338134 4456 scope.go:122] "RemoveContainer" containerID="e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924" Oct 31 03:14:21.338495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:21.338461 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:21.346901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:21.346856 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffa20b5-7f4f-4160-888d-31ae12bda586-kube-api-access-dgtk4" pod "bffa20b5-7f4f-4160-888d-31ae12bda586" (UID: "bffa20b5-7f4f-4160-888d-31ae12bda586"). InnerVolumeSpecName "kube-api-access-dgtk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:14:21.444108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:21.441552 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgtk4\" (UniqueName: \"kubernetes.io/projected/bffa20b5-7f4f-4160-888d-31ae12bda586-kube-api-access-dgtk4\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:14:21.445216 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:21.445191 4456 scope.go:122] "RemoveContainer" containerID="e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924" Oct 31 03:14:21.446006 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:21.445983 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924\": container with ID starting with e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924 not found: ID does not exist" containerID="e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924" Oct 31 03:14:21.446274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:21.446246 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924"} err="failed to get container status \"e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924\": rpc error: code = NotFound desc = could not find container \"e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924\": container with ID starting with e125d691e3fedce5eb314acf168d47ac8be34d1eab082b5a8fff348ea6018924 not found: ID does not exist" Oct 31 03:14:21.652426 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:21.652307 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-a71084\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="bffa20b5-7f4f-4160-888d-31ae12bda586" pod="opendatahub/test-curl-6968-a71084" Oct 31 03:14:22.443425 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:22.443386 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffa20b5-7f4f-4160-888d-31ae12bda586" path="/var/lib/kubelet/pods/bffa20b5-7f4f-4160-888d-31ae12bda586/volumes" Oct 31 03:14:22.720736 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:22.720709 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-11e30e" Oct 31 03:14:22.780601 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:22.780560 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac7217d-3a6d-4fd0-810c-26fe6762d0d5-kube-api-access-m676c\" (UniqueName: \"kubernetes.io/projected/8ac7217d-3a6d-4fd0-810c-26fe6762d0d5-kube-api-access-m676c\") pod \"8ac7217d-3a6d-4fd0-810c-26fe6762d0d5\" (UID: \"8ac7217d-3a6d-4fd0-810c-26fe6762d0d5\") " Oct 31 03:14:22.796141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:22.787184 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac7217d-3a6d-4fd0-810c-26fe6762d0d5-kube-api-access-m676c" pod "8ac7217d-3a6d-4fd0-810c-26fe6762d0d5" (UID: "8ac7217d-3a6d-4fd0-810c-26fe6762d0d5"). InnerVolumeSpecName "kube-api-access-m676c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:14:22.882746 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:22.882702 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m676c\" (UniqueName: \"kubernetes.io/projected/8ac7217d-3a6d-4fd0-810c-26fe6762d0d5-kube-api-access-m676c\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:14:23.375588 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:23.375548 4456 generic.go:356] "Generic (PLEG): container finished" podID="8ac7217d-3a6d-4fd0-810c-26fe6762d0d5" containerID="e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3" exitCode=137 Oct 31 03:14:23.375804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:23.375602 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6968-11e30e" Oct 31 03:14:23.375804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:23.375664 4456 scope.go:122] "RemoveContainer" containerID="e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3" Oct 31 03:14:23.393440 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:23.393297 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6968-11e30e\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="8ac7217d-3a6d-4fd0-810c-26fe6762d0d5" pod="opendatahub/test-curl-6968-11e30e" Oct 31 03:14:23.393519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:23.393452 4456 scope.go:122] "RemoveContainer" containerID="e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3" Oct 31 03:14:23.393763 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:23.393742 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3\": container with ID starting with e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3 not found: ID does not exist" containerID="e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3" Oct 31 03:14:23.393822 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:23.393773 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3"} err="failed to get container status \"e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3\": rpc error: code = NotFound desc = could not find container \"e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3\": container with ID starting with e7b62ff22c0df3d272f4b3732b95312fb8179def6501952e3560c65cd396f9a3 not found: ID does not exist" Oct 31 03:14:24.436728 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:24.436697 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac7217d-3a6d-4fd0-810c-26fe6762d0d5" path="/var/lib/kubelet/pods/8ac7217d-3a6d-4fd0-810c-26fe6762d0d5/volumes" Oct 31 03:14:27.353128 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.353090 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb"] Oct 31 03:14:27.354056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.354013 4456 cpu_manager.go:414] "containerMap: removing container" podUID="8ac7217d-3a6d-4fd0-810c-26fe6762d0d5" containerName="test-curl-6968-11e30e" Oct 31 03:14:27.354056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.354052 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac7217d-3a6d-4fd0-810c-26fe6762d0d5" containerName="test-curl-6968-11e30e" Oct 31 03:14:27.354268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.354242 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac7217d-3a6d-4fd0-810c-26fe6762d0d5" containerName="test-curl-6968-11e30e" Oct 31 03:14:27.359438 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.359407 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:14:27.365908 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.365882 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f1bfbe55\"" Oct 31 03:14:27.366391 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.366372 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd"] Oct 31 03:14:27.370897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.370878 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:14:27.373465 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.373441 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb"] Oct 31 03:14:27.384286 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.384258 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd"] Oct 31 03:14:27.394144 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.394121 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f1bfbe55-dockercfg-xvb2n\"" Oct 31 03:14:27.485779 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.485736 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlbjv\" (UniqueName: \"kubernetes.io/projected/6943cde0-73dd-4c09-b335-531048357951-kube-api-access-hlbjv\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd\" (UID: \"6943cde0-73dd-4c09-b335-531048357951\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:14:27.485973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.485801 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-plugins-config-volume\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb\" (UID: \"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:14:27.485973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.485919 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6943cde0-73dd-4c09-b335-531048357951-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd\" (UID: \"6943cde0-73dd-4c09-b335-531048357951\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:14:27.486274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.486254 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfrbr\" (UniqueName: \"kubernetes.io/projected/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-kube-api-access-lfrbr\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb\" (UID: \"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:14:27.592067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.592009 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6943cde0-73dd-4c09-b335-531048357951-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd\" (UID: \"6943cde0-73dd-4c09-b335-531048357951\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:14:27.593440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.593411 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfrbr\" (UniqueName: \"kubernetes.io/projected/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-kube-api-access-lfrbr\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb\" (UID: \"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:14:27.594024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.594001 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlbjv\" (UniqueName: \"kubernetes.io/projected/6943cde0-73dd-4c09-b335-531048357951-kube-api-access-hlbjv\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd\" (UID: \"6943cde0-73dd-4c09-b335-531048357951\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:14:27.608103 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.594455 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-plugins-config-volume\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb\" (UID: \"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:14:27.611228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.593138 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6943cde0-73dd-4c09-b335-531048357951-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd\" (UID: \"6943cde0-73dd-4c09-b335-531048357951\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:14:27.611404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.610980 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-plugins-config-volume\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb\" (UID: \"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:14:27.623117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.617979 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfrbr\" (UniqueName: \"kubernetes.io/projected/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-kube-api-access-lfrbr\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb\" (UID: \"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:14:27.635015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.634956 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlbjv\" (UniqueName: \"kubernetes.io/projected/6943cde0-73dd-4c09-b335-531048357951-kube-api-access-hlbjv\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd\" (UID: \"6943cde0-73dd-4c09-b335-531048357951\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:14:27.711067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.710385 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:14:27.716848 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:27.716775 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:14:28.121197 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:14:28.121167 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c29c9c_1fd0_47c9_bd79_0bbffb9e3902.slice/crio-511bff099b7c7ea14950b133062bb343e756566a2dfeba66149efbae2a523458 WatchSource:0}: Error finding container 511bff099b7c7ea14950b133062bb343e756566a2dfeba66149efbae2a523458: Status 404 returned error can't find the container with id 511bff099b7c7ea14950b133062bb343e756566a2dfeba66149efbae2a523458 Oct 31 03:14:28.121815 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:14:28.121792 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6943cde0_73dd_4c09_b335_531048357951.slice/crio-48f58e0fe0bb7cb3e3c704ad64ecfa0a625aaeec5abb23a6bfc2001669248788 WatchSource:0}: Error finding container 48f58e0fe0bb7cb3e3c704ad64ecfa0a625aaeec5abb23a6bfc2001669248788: Status 404 returned error can't find the container with id 48f58e0fe0bb7cb3e3c704ad64ecfa0a625aaeec5abb23a6bfc2001669248788 Oct 31 03:14:28.122610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:28.122585 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb"] Oct 31 03:14:28.128223 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:28.128125 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd"] Oct 31 03:14:28.428653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:28.428182 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" event={"ID":"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902","Type":"ContainerStarted","Data":"3634a9b0dd499b352d81f727f10277d392f7e978f3ba33026ff86b2d8307d729"} Oct 31 03:14:28.428653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:28.428279 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:14:28.428653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:28.428384 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" event={"ID":"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902","Type":"ContainerStarted","Data":"511bff099b7c7ea14950b133062bb343e756566a2dfeba66149efbae2a523458"} Oct 31 03:14:28.439388 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:28.439361 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:14:28.439522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:28.439408 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" event={"ID":"6943cde0-73dd-4c09-b335-531048357951","Type":"ContainerStarted","Data":"e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a"} Oct 31 03:14:28.439522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:28.439432 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" event={"ID":"6943cde0-73dd-4c09-b335-531048357951","Type":"ContainerStarted","Data":"48f58e0fe0bb7cb3e3c704ad64ecfa0a625aaeec5abb23a6bfc2001669248788"} Oct 31 03:14:28.449690 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:28.449643 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" podStartSLOduration=1.449629862 podStartE2EDuration="1.449629862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:28.449112468 +0000 UTC m=+1150.129638596" watchObservedRunningTime="2026-10-31 03:14:28.449629862 +0000 UTC m=+1150.130155988" Oct 31 03:14:28.480919 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:28.480856 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" podStartSLOduration=1.480838084 podStartE2EDuration="1.480838084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:28.477063335 +0000 UTC m=+1150.157589463" watchObservedRunningTime="2026-10-31 03:14:28.480838084 +0000 UTC m=+1150.161364212" Oct 31 03:14:32.143231 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:14:32.143190 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:14:39.440877 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:39.440840 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:14:40.458470 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:40.458442 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:14:53.613100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.608827 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt"] Oct 31 03:14:53.613940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.613917 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.638706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.638676 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt"] Oct 31 03:14:53.642116 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.642089 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f1bfbe55\"/\"openshift-service-ca.crt\"" Oct 31 03:14:53.644460 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.644426 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f1bfbe55\"/\"kube-root-ca.crt\"" Oct 31 03:14:53.644587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.644519 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f1bfbe55\"/\"default-dockercfg-tv5wp\"" Oct 31 03:14:53.644656 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.644598 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f1bfbe55\"/\"e2e-lifecycle-model-f1bfbe55-kserve-self-signed-certs\"" Oct 31 03:14:53.704504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.704470 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-model-cache\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.704675 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.704583 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf77w\" (UniqueName: \"kubernetes.io/projected/5f227327-853e-4e94-ad44-ec954ff5ab4b-kube-api-access-sf77w\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.704728 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.704702 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f227327-853e-4e94-ad44-ec954ff5ab4b-tls-certs\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.704783 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.704746 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-dshm\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.704904 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.704881 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-tmp-dir\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.704999 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.704984 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-home\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.810502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.810462 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-model-cache\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.811320 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.811276 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf77w\" (UniqueName: \"kubernetes.io/projected/5f227327-853e-4e94-ad44-ec954ff5ab4b-kube-api-access-sf77w\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.812294 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.812267 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f227327-853e-4e94-ad44-ec954ff5ab4b-tls-certs\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.812558 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.812538 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-dshm\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.812797 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.812770 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-tmp-dir\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.813001 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.812982 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-home\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.818115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.811109 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-model-cache\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.818115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.816574 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-tmp-dir\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.818115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.817147 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-home\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.821750 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.821724 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-dshm\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.829996 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.829939 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf77w\" (UniqueName: \"kubernetes.io/projected/5f227327-853e-4e94-ad44-ec954ff5ab4b-kube-api-access-sf77w\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.850677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.849727 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f227327-853e-4e94-ad44-ec954ff5ab4b-tls-certs\") pod \"e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:53.936324 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:53.936283 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:14:54.260932 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:14:54.260893 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f227327_853e_4e94_ad44_ec954ff5ab4b.slice/crio-f2d98f39f078f9968fd336ac855e18cdb05d41a604c4a35befecdbb62829a390 WatchSource:0}: Error finding container f2d98f39f078f9968fd336ac855e18cdb05d41a604c4a35befecdbb62829a390: Status 404 returned error can't find the container with id f2d98f39f078f9968fd336ac855e18cdb05d41a604c4a35befecdbb62829a390 Oct 31 03:14:54.261992 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:54.261863 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt"] Oct 31 03:14:54.691793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:54.691753 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" event={"ID":"5f227327-853e-4e94-ad44-ec954ff5ab4b","Type":"ContainerStarted","Data":"0d25bc2e76cc6e455f8d0e273d54de693fba7185155052aad19905397480f38c"} Oct 31 03:14:54.692264 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:54.691808 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" event={"ID":"5f227327-853e-4e94-ad44-ec954ff5ab4b","Type":"ContainerStarted","Data":"f2d98f39f078f9968fd336ac855e18cdb05d41a604c4a35befecdbb62829a390"} Oct 31 03:14:54.719990 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:54.719926 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podStartSLOduration=1.719900103 podStartE2EDuration="1.719900103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:54.713460023 +0000 UTC m=+1176.393986151" watchObservedRunningTime="2026-10-31 03:14:54.719900103 +0000 UTC m=+1176.400426231" Oct 31 03:14:54.937308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:14:54.937269 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:15:00.171057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.170902 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d"] Oct 31 03:15:00.190102 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.187355 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" Oct 31 03:15:00.207056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.197613 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:15:00.207056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.198660 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq"] Oct 31 03:15:00.207056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.206652 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" Oct 31 03:15:00.211414 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.211099 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d"] Oct 31 03:15:00.219982 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.219957 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg"] Oct 31 03:15:00.226825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.226805 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" Oct 31 03:15:00.235141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.233247 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159d54c2-29e1-4185-aed6-d7e839d50d4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890275-m578d\" (UID: \"159d54c2-29e1-4185-aed6-d7e839d50d4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" Oct 31 03:15:00.235141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.233362 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq\" (UID: \"895d18c5-e44e-46bb-bee5-f6bac0ec11ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" Oct 31 03:15:00.235141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.233514 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s7gc\" (UniqueName: \"kubernetes.io/projected/159d54c2-29e1-4185-aed6-d7e839d50d4b-kube-api-access-5s7gc\") pod \"maas-api-key-cleanup-29890275-m578d\" (UID: \"159d54c2-29e1-4185-aed6-d7e839d50d4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" Oct 31 03:15:00.235141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.233566 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscf8\" (UniqueName: \"kubernetes.io/projected/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-kube-api-access-kscf8\") pod \"maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq\" (UID: \"895d18c5-e44e-46bb-bee5-f6bac0ec11ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" Oct 31 03:15:00.248074 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.244579 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz"] Oct 31 03:15:00.253073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.253010 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq"] Oct 31 03:15:00.253347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.253330 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" Oct 31 03:15:00.264692 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.264616 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg"] Oct 31 03:15:00.269430 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.269355 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz"] Oct 31 03:15:00.280101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.276455 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx"] Oct 31 03:15:00.283225 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.282486 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" Oct 31 03:15:00.314524 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.314460 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx"] Oct 31 03:15:00.334654 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.334625 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq\" (UID: \"895d18c5-e44e-46bb-bee5-f6bac0ec11ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" Oct 31 03:15:00.334810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.334688 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fq8\" (UniqueName: \"kubernetes.io/projected/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-kube-api-access-x6fq8\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg\" (UID: \"94bb5800-77c6-47a9-a7b4-ae82a1d2e074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" Oct 31 03:15:00.334810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.334765 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58936d3-d580-4c84-9d7f-59502f7312b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz\" (UID: \"e58936d3-d580-4c84-9d7f-59502f7312b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" Oct 31 03:15:00.334897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.334876 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s7gc\" (UniqueName: \"kubernetes.io/projected/159d54c2-29e1-4185-aed6-d7e839d50d4b-kube-api-access-5s7gc\") pod \"maas-api-key-cleanup-29890275-m578d\" (UID: \"159d54c2-29e1-4185-aed6-d7e839d50d4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" Oct 31 03:15:00.334957 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.334936 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscf8\" (UniqueName: \"kubernetes.io/projected/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-kube-api-access-kscf8\") pod \"maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq\" (UID: \"895d18c5-e44e-46bb-bee5-f6bac0ec11ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" Oct 31 03:15:00.335022 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.335001 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hztqv\" (UniqueName: \"kubernetes.io/projected/271d05c8-13c6-47ff-95d8-13d3653f9a3e-kube-api-access-hztqv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx\" (UID: \"271d05c8-13c6-47ff-95d8-13d3653f9a3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" Oct 31 03:15:00.335155 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.335115 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg\" (UID: \"94bb5800-77c6-47a9-a7b4-ae82a1d2e074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" Oct 31 03:15:00.335242 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.335211 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/271d05c8-13c6-47ff-95d8-13d3653f9a3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx\" (UID: \"271d05c8-13c6-47ff-95d8-13d3653f9a3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" Oct 31 03:15:00.335313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.335241 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbwz\" (UniqueName: \"kubernetes.io/projected/e58936d3-d580-4c84-9d7f-59502f7312b1-kube-api-access-xlbwz\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz\" (UID: \"e58936d3-d580-4c84-9d7f-59502f7312b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" Oct 31 03:15:00.335313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.335287 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159d54c2-29e1-4185-aed6-d7e839d50d4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890275-m578d\" (UID: \"159d54c2-29e1-4185-aed6-d7e839d50d4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" Oct 31 03:15:00.335524 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.335403 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq\" (UID: \"895d18c5-e44e-46bb-bee5-f6bac0ec11ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" Oct 31 03:15:00.335973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.335950 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159d54c2-29e1-4185-aed6-d7e839d50d4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890275-m578d\" (UID: \"159d54c2-29e1-4185-aed6-d7e839d50d4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" Oct 31 03:15:00.351956 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.351925 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s7gc\" (UniqueName: \"kubernetes.io/projected/159d54c2-29e1-4185-aed6-d7e839d50d4b-kube-api-access-5s7gc\") pod \"maas-api-key-cleanup-29890275-m578d\" (UID: \"159d54c2-29e1-4185-aed6-d7e839d50d4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" Oct 31 03:15:00.352173 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.351925 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscf8\" (UniqueName: \"kubernetes.io/projected/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-kube-api-access-kscf8\") pod \"maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq\" (UID: \"895d18c5-e44e-46bb-bee5-f6bac0ec11ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" Oct 31 03:15:00.437026 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.436937 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58936d3-d580-4c84-9d7f-59502f7312b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz\" (UID: \"e58936d3-d580-4c84-9d7f-59502f7312b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" Oct 31 03:15:00.437026 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.437023 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hztqv\" (UniqueName: \"kubernetes.io/projected/271d05c8-13c6-47ff-95d8-13d3653f9a3e-kube-api-access-hztqv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx\" (UID: \"271d05c8-13c6-47ff-95d8-13d3653f9a3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" Oct 31 03:15:00.437267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.437117 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg\" (UID: \"94bb5800-77c6-47a9-a7b4-ae82a1d2e074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" Oct 31 03:15:00.437267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.437187 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/271d05c8-13c6-47ff-95d8-13d3653f9a3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx\" (UID: \"271d05c8-13c6-47ff-95d8-13d3653f9a3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" Oct 31 03:15:00.437267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.437215 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbwz\" (UniqueName: \"kubernetes.io/projected/e58936d3-d580-4c84-9d7f-59502f7312b1-kube-api-access-xlbwz\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz\" (UID: \"e58936d3-d580-4c84-9d7f-59502f7312b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" Oct 31 03:15:00.437439 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.437356 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fq8\" (UniqueName: \"kubernetes.io/projected/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-kube-api-access-x6fq8\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg\" (UID: \"94bb5800-77c6-47a9-a7b4-ae82a1d2e074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" Oct 31 03:15:00.437706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.437679 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58936d3-d580-4c84-9d7f-59502f7312b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz\" (UID: \"e58936d3-d580-4c84-9d7f-59502f7312b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" Oct 31 03:15:00.437870 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.437850 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg\" (UID: \"94bb5800-77c6-47a9-a7b4-ae82a1d2e074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" Oct 31 03:15:00.437935 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.437892 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/271d05c8-13c6-47ff-95d8-13d3653f9a3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx\" (UID: \"271d05c8-13c6-47ff-95d8-13d3653f9a3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" Oct 31 03:15:00.450742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.450712 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fq8\" (UniqueName: \"kubernetes.io/projected/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-kube-api-access-x6fq8\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg\" (UID: \"94bb5800-77c6-47a9-a7b4-ae82a1d2e074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" Oct 31 03:15:00.450965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.450943 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbwz\" (UniqueName: \"kubernetes.io/projected/e58936d3-d580-4c84-9d7f-59502f7312b1-kube-api-access-xlbwz\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz\" (UID: \"e58936d3-d580-4c84-9d7f-59502f7312b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" Oct 31 03:15:00.451522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.451503 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hztqv\" (UniqueName: \"kubernetes.io/projected/271d05c8-13c6-47ff-95d8-13d3653f9a3e-kube-api-access-hztqv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx\" (UID: \"271d05c8-13c6-47ff-95d8-13d3653f9a3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" Oct 31 03:15:00.531769 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.531740 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" Oct 31 03:15:00.583479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.583451 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" Oct 31 03:15:00.604901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.604875 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" Oct 31 03:15:00.611218 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.611146 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" Oct 31 03:15:00.622265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.622066 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" Oct 31 03:15:00.875075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:00.866900 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d"] Oct 31 03:15:01.066847 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:15:01.066814 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271d05c8_13c6_47ff_95d8_13d3653f9a3e.slice/crio-34bd34b3a1c20c9f559e87733d698843dc45ff7bac766ac6bab2e0d49fd597ca WatchSource:0}: Error finding container 34bd34b3a1c20c9f559e87733d698843dc45ff7bac766ac6bab2e0d49fd597ca: Status 404 returned error can't find the container with id 34bd34b3a1c20c9f559e87733d698843dc45ff7bac766ac6bab2e0d49fd597ca Oct 31 03:15:01.069821 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.069786 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx"] Oct 31 03:15:01.122995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.122954 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq"] Oct 31 03:15:01.124748 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:15:01.124723 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895d18c5_e44e_46bb_bee5_f6bac0ec11ae.slice/crio-5efb1958fb4e880198315e73a5955c7d9a92e7d8d91413d279f7c15025fa85d2 WatchSource:0}: Error finding container 5efb1958fb4e880198315e73a5955c7d9a92e7d8d91413d279f7c15025fa85d2: Status 404 returned error can't find the container with id 5efb1958fb4e880198315e73a5955c7d9a92e7d8d91413d279f7c15025fa85d2 Oct 31 03:15:01.190647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.190571 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz"] Oct 31 03:15:01.198469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.198437 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg"] Oct 31 03:15:01.360226 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:15:01.360191 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58936d3_d580_4c84_9d7f_59502f7312b1.slice/crio-9b18d767001f481470637ed58ed4eaa37f8e56a8f034fa30efe1a72b1e974bd8 WatchSource:0}: Error finding container 9b18d767001f481470637ed58ed4eaa37f8e56a8f034fa30efe1a72b1e974bd8: Status 404 returned error can't find the container with id 9b18d767001f481470637ed58ed4eaa37f8e56a8f034fa30efe1a72b1e974bd8 Oct 31 03:15:01.762549 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.762514 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" event={"ID":"e58936d3-d580-4c84-9d7f-59502f7312b1","Type":"ContainerStarted","Data":"9b18d767001f481470637ed58ed4eaa37f8e56a8f034fa30efe1a72b1e974bd8"} Oct 31 03:15:01.764498 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.764471 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" event={"ID":"94bb5800-77c6-47a9-a7b4-ae82a1d2e074","Type":"ContainerStarted","Data":"6d120befd12508f68954e29da6dfa24882b2b4894b127512fef90868ba62f294"} Oct 31 03:15:01.767582 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.766724 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" event={"ID":"159d54c2-29e1-4185-aed6-d7e839d50d4b","Type":"ContainerStarted","Data":"a15db89ddc7fcc72ce0104a1e4a22943c002c18eaef3359cff2b5896646585d8"} Oct 31 03:15:01.767582 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.766788 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" event={"ID":"159d54c2-29e1-4185-aed6-d7e839d50d4b","Type":"ContainerStarted","Data":"51b478e8dad1bbddc9d9019f99863cfeb40cde94e21fbb3dc5c9e9d56d76afed"} Oct 31 03:15:01.769977 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.769949 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" event={"ID":"895d18c5-e44e-46bb-bee5-f6bac0ec11ae","Type":"ContainerStarted","Data":"5efb1958fb4e880198315e73a5955c7d9a92e7d8d91413d279f7c15025fa85d2"} Oct 31 03:15:01.772616 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.771946 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" event={"ID":"271d05c8-13c6-47ff-95d8-13d3653f9a3e","Type":"ContainerStarted","Data":"9c3444686df0486f9b9fd172c9394de2f7e1eb4f548f8829ae406a0d108d9702"} Oct 31 03:15:01.772616 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.772029 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" event={"ID":"271d05c8-13c6-47ff-95d8-13d3653f9a3e","Type":"ContainerStarted","Data":"34bd34b3a1c20c9f559e87733d698843dc45ff7bac766ac6bab2e0d49fd597ca"} Oct 31 03:15:01.795580 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.795531 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" podStartSLOduration=1.795516592 podStartE2EDuration="1.795516592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:01.791677146 +0000 UTC m=+1183.472203272" watchObservedRunningTime="2026-10-31 03:15:01.795516592 +0000 UTC m=+1183.476042718" Oct 31 03:15:01.814379 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:01.814321 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" podStartSLOduration=1.8143052929999999 podStartE2EDuration="1.814305293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:01.8142115 +0000 UTC m=+1183.494737628" watchObservedRunningTime="2026-10-31 03:15:01.814305293 +0000 UTC m=+1183.494831424" Oct 31 03:15:02.138559 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:02.138517 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:15:02.785239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.785191 4456 generic.go:356] "Generic (PLEG): container finished" podID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerID="a15db89ddc7fcc72ce0104a1e4a22943c002c18eaef3359cff2b5896646585d8" exitCode=60 Oct 31 03:15:02.785705 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.785442 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" event={"ID":"159d54c2-29e1-4185-aed6-d7e839d50d4b","Type":"ContainerDied","Data":"a15db89ddc7fcc72ce0104a1e4a22943c002c18eaef3359cff2b5896646585d8"} Oct 31 03:15:02.785789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.785767 4456 scope.go:122] "RemoveContainer" containerID="a15db89ddc7fcc72ce0104a1e4a22943c002c18eaef3359cff2b5896646585d8" Oct 31 03:15:02.789744 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.789714 4456 generic.go:356] "Generic (PLEG): container finished" podID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerID="4239fa51c9178eecbb0cdc5f9f72eabaab63b1d941c34bfbd94dbf131cfde973" exitCode=60 Oct 31 03:15:02.789844 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.789778 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" event={"ID":"895d18c5-e44e-46bb-bee5-f6bac0ec11ae","Type":"ContainerDied","Data":"4239fa51c9178eecbb0cdc5f9f72eabaab63b1d941c34bfbd94dbf131cfde973"} Oct 31 03:15:02.790120 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.790098 4456 scope.go:122] "RemoveContainer" containerID="4239fa51c9178eecbb0cdc5f9f72eabaab63b1d941c34bfbd94dbf131cfde973" Oct 31 03:15:02.793652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.793626 4456 generic.go:356] "Generic (PLEG): container finished" podID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerID="9c3444686df0486f9b9fd172c9394de2f7e1eb4f548f8829ae406a0d108d9702" exitCode=60 Oct 31 03:15:02.793865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.793689 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" event={"ID":"271d05c8-13c6-47ff-95d8-13d3653f9a3e","Type":"ContainerDied","Data":"9c3444686df0486f9b9fd172c9394de2f7e1eb4f548f8829ae406a0d108d9702"} Oct 31 03:15:02.793973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.793946 4456 scope.go:122] "RemoveContainer" containerID="9c3444686df0486f9b9fd172c9394de2f7e1eb4f548f8829ae406a0d108d9702" Oct 31 03:15:02.796286 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.796047 4456 generic.go:356] "Generic (PLEG): container finished" podID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerID="ec97a9aaee560b252daa58970a44ea96745e3b297b6805304788ea93ac931f88" exitCode=60 Oct 31 03:15:02.796286 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.796154 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" event={"ID":"e58936d3-d580-4c84-9d7f-59502f7312b1","Type":"ContainerDied","Data":"ec97a9aaee560b252daa58970a44ea96745e3b297b6805304788ea93ac931f88"} Oct 31 03:15:02.796425 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.796356 4456 scope.go:122] "RemoveContainer" containerID="ec97a9aaee560b252daa58970a44ea96745e3b297b6805304788ea93ac931f88" Oct 31 03:15:02.798088 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.798065 4456 generic.go:356] "Generic (PLEG): container finished" podID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerID="725cbd79fb38942a344edb8c9cc7e284dda60213da2f96b1840b5a7b5995a2e8" exitCode=60 Oct 31 03:15:02.798165 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.798118 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" event={"ID":"94bb5800-77c6-47a9-a7b4-ae82a1d2e074","Type":"ContainerDied","Data":"725cbd79fb38942a344edb8c9cc7e284dda60213da2f96b1840b5a7b5995a2e8"} Oct 31 03:15:02.798361 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:02.798342 4456 scope.go:122] "RemoveContainer" containerID="725cbd79fb38942a344edb8c9cc7e284dda60213da2f96b1840b5a7b5995a2e8" Oct 31 03:15:03.820206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:03.820158 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" event={"ID":"e58936d3-d580-4c84-9d7f-59502f7312b1","Type":"ContainerStarted","Data":"d893537dd0a2a22975bf9fc2dc546e4a2b51ea202bb44c00e312428eecd3037b"} Oct 31 03:15:03.823561 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:03.823528 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" event={"ID":"94bb5800-77c6-47a9-a7b4-ae82a1d2e074","Type":"ContainerStarted","Data":"c24e53a3ef999d5891ed9fb5f46efea1bac81d7a8ef10f60ed4f29c2a8de938d"} Oct 31 03:15:03.827295 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:03.827269 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" event={"ID":"159d54c2-29e1-4185-aed6-d7e839d50d4b","Type":"ContainerStarted","Data":"31198f27fe14cc57b9d5966ab2cc0b81f2536fdcdba912fe06b6ad70e397240a"} Oct 31 03:15:03.829562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:03.829538 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" event={"ID":"895d18c5-e44e-46bb-bee5-f6bac0ec11ae","Type":"ContainerStarted","Data":"a93bd0187cebb9d74fb465445157039820684dfe14ae26e43a56121b6e399a83"} Oct 31 03:15:03.831360 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:03.831323 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" event={"ID":"271d05c8-13c6-47ff-95d8-13d3653f9a3e","Type":"ContainerStarted","Data":"211bfc8a663ec55f47e7d2d2d02207060c198cb0e7207c8c8e218e05f552dfae"} Oct 31 03:15:03.843426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:03.843383 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" podStartSLOduration=3.843371318 podStartE2EDuration="3.843371318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:03.842146003 +0000 UTC m=+1185.522672137" watchObservedRunningTime="2026-10-31 03:15:03.843371318 +0000 UTC m=+1185.523897449" Oct 31 03:15:03.876880 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:03.876819 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" podStartSLOduration=3.876801953 podStartE2EDuration="3.876801953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:03.870398592 +0000 UTC m=+1185.550924719" watchObservedRunningTime="2026-10-31 03:15:03.876801953 +0000 UTC m=+1185.557328081" Oct 31 03:15:03.917667 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:03.917615 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" podStartSLOduration=3.917602473 podStartE2EDuration="3.917602473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:03.91383978 +0000 UTC m=+1185.594365906" watchObservedRunningTime="2026-10-31 03:15:03.917602473 +0000 UTC m=+1185.598128599" Oct 31 03:15:03.937497 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:03.937459 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:15:03.954426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:03.954385 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:15:04.842394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.842359 4456 generic.go:356] "Generic (PLEG): container finished" podID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerID="a93bd0187cebb9d74fb465445157039820684dfe14ae26e43a56121b6e399a83" exitCode=60 Oct 31 03:15:04.842891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.842458 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" event={"ID":"895d18c5-e44e-46bb-bee5-f6bac0ec11ae","Type":"ContainerDied","Data":"a93bd0187cebb9d74fb465445157039820684dfe14ae26e43a56121b6e399a83"} Oct 31 03:15:04.842891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.842530 4456 scope.go:122] "RemoveContainer" containerID="4239fa51c9178eecbb0cdc5f9f72eabaab63b1d941c34bfbd94dbf131cfde973" Oct 31 03:15:04.842891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.842781 4456 scope.go:122] "RemoveContainer" containerID="a93bd0187cebb9d74fb465445157039820684dfe14ae26e43a56121b6e399a83" Oct 31 03:15:04.843161 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:04.843141 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq_odh-ai-gateway-infra(895d18c5-e44e-46bb-bee5-f6bac0ec11ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" Oct 31 03:15:04.845310 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.845277 4456 generic.go:356] "Generic (PLEG): container finished" podID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerID="211bfc8a663ec55f47e7d2d2d02207060c198cb0e7207c8c8e218e05f552dfae" exitCode=60 Oct 31 03:15:04.845437 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.845367 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" event={"ID":"271d05c8-13c6-47ff-95d8-13d3653f9a3e","Type":"ContainerDied","Data":"211bfc8a663ec55f47e7d2d2d02207060c198cb0e7207c8c8e218e05f552dfae"} Oct 31 03:15:04.845656 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.845640 4456 scope.go:122] "RemoveContainer" containerID="211bfc8a663ec55f47e7d2d2d02207060c198cb0e7207c8c8e218e05f552dfae" Oct 31 03:15:04.845990 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:04.845965 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx_odh-ai-gateway-infra(271d05c8-13c6-47ff-95d8-13d3653f9a3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" Oct 31 03:15:04.847991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.847969 4456 generic.go:356] "Generic (PLEG): container finished" podID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerID="d893537dd0a2a22975bf9fc2dc546e4a2b51ea202bb44c00e312428eecd3037b" exitCode=60 Oct 31 03:15:04.848121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.848098 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" event={"ID":"e58936d3-d580-4c84-9d7f-59502f7312b1","Type":"ContainerDied","Data":"d893537dd0a2a22975bf9fc2dc546e4a2b51ea202bb44c00e312428eecd3037b"} Oct 31 03:15:04.848435 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.848419 4456 scope.go:122] "RemoveContainer" containerID="d893537dd0a2a22975bf9fc2dc546e4a2b51ea202bb44c00e312428eecd3037b" Oct 31 03:15:04.848707 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:04.848688 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz_odh-ai-gateway-infra(e58936d3-d580-4c84-9d7f-59502f7312b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" Oct 31 03:15:04.850682 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.850657 4456 generic.go:356] "Generic (PLEG): container finished" podID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerID="c24e53a3ef999d5891ed9fb5f46efea1bac81d7a8ef10f60ed4f29c2a8de938d" exitCode=60 Oct 31 03:15:04.850812 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.850796 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" event={"ID":"94bb5800-77c6-47a9-a7b4-ae82a1d2e074","Type":"ContainerDied","Data":"c24e53a3ef999d5891ed9fb5f46efea1bac81d7a8ef10f60ed4f29c2a8de938d"} Oct 31 03:15:04.851190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.851169 4456 scope.go:122] "RemoveContainer" containerID="c24e53a3ef999d5891ed9fb5f46efea1bac81d7a8ef10f60ed4f29c2a8de938d" Oct 31 03:15:04.851499 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:04.851481 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg_odh-ai-gateway-infra(94bb5800-77c6-47a9-a7b4-ae82a1d2e074)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" Oct 31 03:15:04.853642 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.853621 4456 generic.go:356] "Generic (PLEG): container finished" podID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerID="31198f27fe14cc57b9d5966ab2cc0b81f2536fdcdba912fe06b6ad70e397240a" exitCode=60 Oct 31 03:15:04.854661 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.854646 4456 scope.go:122] "RemoveContainer" containerID="31198f27fe14cc57b9d5966ab2cc0b81f2536fdcdba912fe06b6ad70e397240a" Oct 31 03:15:04.854917 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:04.854899 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890275-m578d_odh-ai-gateway-infra(159d54c2-29e1-4185-aed6-d7e839d50d4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" Oct 31 03:15:04.854956 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.854934 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" event={"ID":"159d54c2-29e1-4185-aed6-d7e839d50d4b","Type":"ContainerDied","Data":"31198f27fe14cc57b9d5966ab2cc0b81f2536fdcdba912fe06b6ad70e397240a"} Oct 31 03:15:04.860211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.860187 4456 scope.go:122] "RemoveContainer" containerID="9c3444686df0486f9b9fd172c9394de2f7e1eb4f548f8829ae406a0d108d9702" Oct 31 03:15:04.868446 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.868421 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:15:04.879676 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.879660 4456 scope.go:122] "RemoveContainer" containerID="ec97a9aaee560b252daa58970a44ea96745e3b297b6805304788ea93ac931f88" Oct 31 03:15:04.903745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.903684 4456 scope.go:122] "RemoveContainer" containerID="725cbd79fb38942a344edb8c9cc7e284dda60213da2f96b1840b5a7b5995a2e8" Oct 31 03:15:04.938130 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:04.938098 4456 scope.go:122] "RemoveContainer" containerID="a15db89ddc7fcc72ce0104a1e4a22943c002c18eaef3359cff2b5896646585d8" Oct 31 03:15:05.914227 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:05.914198 4456 scope.go:122] "RemoveContainer" containerID="211bfc8a663ec55f47e7d2d2d02207060c198cb0e7207c8c8e218e05f552dfae" Oct 31 03:15:05.915409 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:05.915160 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx_odh-ai-gateway-infra(271d05c8-13c6-47ff-95d8-13d3653f9a3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" Oct 31 03:15:05.916431 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:05.916415 4456 scope.go:122] "RemoveContainer" containerID="d893537dd0a2a22975bf9fc2dc546e4a2b51ea202bb44c00e312428eecd3037b" Oct 31 03:15:05.916926 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:05.916905 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz_odh-ai-gateway-infra(e58936d3-d580-4c84-9d7f-59502f7312b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" Oct 31 03:15:05.918503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:05.918482 4456 scope.go:122] "RemoveContainer" containerID="c24e53a3ef999d5891ed9fb5f46efea1bac81d7a8ef10f60ed4f29c2a8de938d" Oct 31 03:15:05.918819 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:05.918777 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg_odh-ai-gateway-infra(94bb5800-77c6-47a9-a7b4-ae82a1d2e074)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" Oct 31 03:15:05.921297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:05.921280 4456 scope.go:122] "RemoveContainer" containerID="31198f27fe14cc57b9d5966ab2cc0b81f2536fdcdba912fe06b6ad70e397240a" Oct 31 03:15:05.921601 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:05.921563 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890275-m578d_odh-ai-gateway-infra(159d54c2-29e1-4185-aed6-d7e839d50d4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" Oct 31 03:15:05.924113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:05.924091 4456 scope.go:122] "RemoveContainer" containerID="a93bd0187cebb9d74fb465445157039820684dfe14ae26e43a56121b6e399a83" Oct 31 03:15:05.924377 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:05.924355 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq_odh-ai-gateway-infra(895d18c5-e44e-46bb-bee5-f6bac0ec11ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" Oct 31 03:15:15.428432 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:15.428399 4456 scope.go:122] "RemoveContainer" containerID="211bfc8a663ec55f47e7d2d2d02207060c198cb0e7207c8c8e218e05f552dfae" Oct 31 03:15:15.428917 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:15.428552 4456 scope.go:122] "RemoveContainer" containerID="d893537dd0a2a22975bf9fc2dc546e4a2b51ea202bb44c00e312428eecd3037b" Oct 31 03:15:15.428917 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:15.428908 4456 scope.go:122] "RemoveContainer" containerID="c24e53a3ef999d5891ed9fb5f46efea1bac81d7a8ef10f60ed4f29c2a8de938d" Oct 31 03:15:16.018068 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:16.017952 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" event={"ID":"e58936d3-d580-4c84-9d7f-59502f7312b1","Type":"ContainerStarted","Data":"34c57cd3a45f9ed9126225f5981f502c3b60eb9a331c88592e8b3c46f5804a6f"} Oct 31 03:15:16.020497 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:16.020464 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" event={"ID":"94bb5800-77c6-47a9-a7b4-ae82a1d2e074","Type":"ContainerStarted","Data":"596532fc271d79b939cd1ee338998fde4fcb86312d58c3e98cc5d3c0e15d64e3"} Oct 31 03:15:16.429748 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:16.429713 4456 scope.go:122] "RemoveContainer" containerID="a93bd0187cebb9d74fb465445157039820684dfe14ae26e43a56121b6e399a83" Oct 31 03:15:17.031471 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.031374 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" event={"ID":"895d18c5-e44e-46bb-bee5-f6bac0ec11ae","Type":"ContainerStarted","Data":"a3e8cbd1c42a0a5d791c55f0ba7b9bac415e37b4fd6f62f66cf07ac57efd77f9"} Oct 31 03:15:17.033402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.033374 4456 generic.go:356] "Generic (PLEG): container finished" podID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerID="a435c2bb43015cfc5591783986a8588d83646c11254caae742c40968dd9d7a5c" exitCode=60 Oct 31 03:15:17.033552 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.033452 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" event={"ID":"271d05c8-13c6-47ff-95d8-13d3653f9a3e","Type":"ContainerDied","Data":"a435c2bb43015cfc5591783986a8588d83646c11254caae742c40968dd9d7a5c"} Oct 31 03:15:17.033552 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.033513 4456 scope.go:122] "RemoveContainer" containerID="211bfc8a663ec55f47e7d2d2d02207060c198cb0e7207c8c8e218e05f552dfae" Oct 31 03:15:17.033895 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.033877 4456 scope.go:122] "RemoveContainer" containerID="a435c2bb43015cfc5591783986a8588d83646c11254caae742c40968dd9d7a5c" Oct 31 03:15:17.034255 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:17.034220 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx_odh-ai-gateway-infra(271d05c8-13c6-47ff-95d8-13d3653f9a3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" Oct 31 03:15:17.037003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.036981 4456 generic.go:356] "Generic (PLEG): container finished" podID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerID="34c57cd3a45f9ed9126225f5981f502c3b60eb9a331c88592e8b3c46f5804a6f" exitCode=60 Oct 31 03:15:17.037168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.037087 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" event={"ID":"e58936d3-d580-4c84-9d7f-59502f7312b1","Type":"ContainerDied","Data":"34c57cd3a45f9ed9126225f5981f502c3b60eb9a331c88592e8b3c46f5804a6f"} Oct 31 03:15:17.037405 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.037390 4456 scope.go:122] "RemoveContainer" containerID="34c57cd3a45f9ed9126225f5981f502c3b60eb9a331c88592e8b3c46f5804a6f" Oct 31 03:15:17.037702 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:17.037678 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz_odh-ai-gateway-infra(e58936d3-d580-4c84-9d7f-59502f7312b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" Oct 31 03:15:17.039199 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.039170 4456 generic.go:356] "Generic (PLEG): container finished" podID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerID="596532fc271d79b939cd1ee338998fde4fcb86312d58c3e98cc5d3c0e15d64e3" exitCode=60 Oct 31 03:15:17.039285 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.039235 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" event={"ID":"94bb5800-77c6-47a9-a7b4-ae82a1d2e074","Type":"ContainerDied","Data":"596532fc271d79b939cd1ee338998fde4fcb86312d58c3e98cc5d3c0e15d64e3"} Oct 31 03:15:17.039590 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.039571 4456 scope.go:122] "RemoveContainer" containerID="596532fc271d79b939cd1ee338998fde4fcb86312d58c3e98cc5d3c0e15d64e3" Oct 31 03:15:17.039849 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:17.039831 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg_odh-ai-gateway-infra(94bb5800-77c6-47a9-a7b4-ae82a1d2e074)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" Oct 31 03:15:17.051987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.051710 4456 scope.go:122] "RemoveContainer" containerID="d893537dd0a2a22975bf9fc2dc546e4a2b51ea202bb44c00e312428eecd3037b" Oct 31 03:15:17.054159 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.054093 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz"] Oct 31 03:15:17.067369 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.067350 4456 scope.go:122] "RemoveContainer" containerID="c24e53a3ef999d5891ed9fb5f46efea1bac81d7a8ef10f60ed4f29c2a8de938d" Oct 31 03:15:17.076013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:17.075993 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg"] Oct 31 03:15:18.052243 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.052208 4456 generic.go:356] "Generic (PLEG): container finished" podID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerID="a3e8cbd1c42a0a5d791c55f0ba7b9bac415e37b4fd6f62f66cf07ac57efd77f9" exitCode=60 Oct 31 03:15:18.052612 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.052307 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" event={"ID":"895d18c5-e44e-46bb-bee5-f6bac0ec11ae","Type":"ContainerDied","Data":"a3e8cbd1c42a0a5d791c55f0ba7b9bac415e37b4fd6f62f66cf07ac57efd77f9"} Oct 31 03:15:18.052612 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.052383 4456 scope.go:122] "RemoveContainer" containerID="a93bd0187cebb9d74fb465445157039820684dfe14ae26e43a56121b6e399a83" Oct 31 03:15:18.052793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.052774 4456 scope.go:122] "RemoveContainer" containerID="a3e8cbd1c42a0a5d791c55f0ba7b9bac415e37b4fd6f62f66cf07ac57efd77f9" Oct 31 03:15:18.053195 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:18.053168 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq_odh-ai-gateway-infra(895d18c5-e44e-46bb-bee5-f6bac0ec11ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" Oct 31 03:15:18.066564 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.066537 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq"] Oct 31 03:15:18.217379 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.211504 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx"] Oct 31 03:15:18.389048 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.389006 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" Oct 31 03:15:18.435400 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.435268 4456 scope.go:122] "RemoveContainer" containerID="31198f27fe14cc57b9d5966ab2cc0b81f2536fdcdba912fe06b6ad70e397240a" Oct 31 03:15:18.435757 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.435732 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" Oct 31 03:15:18.492940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.492910 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-kube-api-access-x6fq8\" (UniqueName: \"kubernetes.io/projected/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-kube-api-access-x6fq8\") pod \"94bb5800-77c6-47a9-a7b4-ae82a1d2e074\" (UID: \"94bb5800-77c6-47a9-a7b4-ae82a1d2e074\") " Oct 31 03:15:18.493108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.492995 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-maas-api-service-ca\") pod \"94bb5800-77c6-47a9-a7b4-ae82a1d2e074\" (UID: \"94bb5800-77c6-47a9-a7b4-ae82a1d2e074\") " Oct 31 03:15:18.493375 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.493337 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-maas-api-service-ca" pod "94bb5800-77c6-47a9-a7b4-ae82a1d2e074" (UID: "94bb5800-77c6-47a9-a7b4-ae82a1d2e074"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:18.494235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.494214 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-maas-api-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.498982 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.498387 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-kube-api-access-x6fq8" pod "94bb5800-77c6-47a9-a7b4-ae82a1d2e074" (UID: "94bb5800-77c6-47a9-a7b4-ae82a1d2e074"). InnerVolumeSpecName "kube-api-access-x6fq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:18.528198 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.528173 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" Oct 31 03:15:18.595845 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.595754 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271d05c8-13c6-47ff-95d8-13d3653f9a3e-kube-api-access-hztqv\" (UniqueName: \"kubernetes.io/projected/271d05c8-13c6-47ff-95d8-13d3653f9a3e-kube-api-access-hztqv\") pod \"271d05c8-13c6-47ff-95d8-13d3653f9a3e\" (UID: \"271d05c8-13c6-47ff-95d8-13d3653f9a3e\") " Oct 31 03:15:18.596009 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.595881 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58936d3-d580-4c84-9d7f-59502f7312b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58936d3-d580-4c84-9d7f-59502f7312b1-maas-api-service-ca\") pod \"e58936d3-d580-4c84-9d7f-59502f7312b1\" (UID: \"e58936d3-d580-4c84-9d7f-59502f7312b1\") " Oct 31 03:15:18.596009 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.595998 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271d05c8-13c6-47ff-95d8-13d3653f9a3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/271d05c8-13c6-47ff-95d8-13d3653f9a3e-maas-api-service-ca\") pod \"271d05c8-13c6-47ff-95d8-13d3653f9a3e\" (UID: \"271d05c8-13c6-47ff-95d8-13d3653f9a3e\") " Oct 31 03:15:18.596147 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.596099 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58936d3-d580-4c84-9d7f-59502f7312b1-kube-api-access-xlbwz\" (UniqueName: \"kubernetes.io/projected/e58936d3-d580-4c84-9d7f-59502f7312b1-kube-api-access-xlbwz\") pod \"e58936d3-d580-4c84-9d7f-59502f7312b1\" (UID: \"e58936d3-d580-4c84-9d7f-59502f7312b1\") " Oct 31 03:15:18.596524 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.596493 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58936d3-d580-4c84-9d7f-59502f7312b1-maas-api-service-ca" pod "e58936d3-d580-4c84-9d7f-59502f7312b1" (UID: "e58936d3-d580-4c84-9d7f-59502f7312b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:18.596524 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.596496 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271d05c8-13c6-47ff-95d8-13d3653f9a3e-maas-api-service-ca" pod "271d05c8-13c6-47ff-95d8-13d3653f9a3e" (UID: "271d05c8-13c6-47ff-95d8-13d3653f9a3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:18.596939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.596915 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6fq8\" (UniqueName: \"kubernetes.io/projected/94bb5800-77c6-47a9-a7b4-ae82a1d2e074-kube-api-access-x6fq8\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.597024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.596945 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58936d3-d580-4c84-9d7f-59502f7312b1-maas-api-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.597024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.597005 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/271d05c8-13c6-47ff-95d8-13d3653f9a3e-maas-api-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.647391 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.647358 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271d05c8-13c6-47ff-95d8-13d3653f9a3e-kube-api-access-hztqv" pod "271d05c8-13c6-47ff-95d8-13d3653f9a3e" (UID: "271d05c8-13c6-47ff-95d8-13d3653f9a3e"). InnerVolumeSpecName "kube-api-access-hztqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:18.647539 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.647403 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58936d3-d580-4c84-9d7f-59502f7312b1-kube-api-access-xlbwz" pod "e58936d3-d580-4c84-9d7f-59502f7312b1" (UID: "e58936d3-d580-4c84-9d7f-59502f7312b1"). InnerVolumeSpecName "kube-api-access-xlbwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:18.698343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.698313 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hztqv\" (UniqueName: \"kubernetes.io/projected/271d05c8-13c6-47ff-95d8-13d3653f9a3e-kube-api-access-hztqv\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.698343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.698341 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlbwz\" (UniqueName: \"kubernetes.io/projected/e58936d3-d580-4c84-9d7f-59502f7312b1-kube-api-access-xlbwz\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.761608 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:18.761569 4456 scope.go:122] "RemoveContainer" containerID="1074d8e1d1f7f22ddd2df32f2b5deb257fd2a1189589e5387cf1f4e391b431b9" Oct 31 03:15:19.077012 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.076978 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" Oct 31 03:15:19.078973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.077004 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx" event={"ID":"271d05c8-13c6-47ff-95d8-13d3653f9a3e","Type":"ContainerDied","Data":"34bd34b3a1c20c9f559e87733d698843dc45ff7bac766ac6bab2e0d49fd597ca"} Oct 31 03:15:19.078973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.077143 4456 scope.go:122] "RemoveContainer" containerID="a435c2bb43015cfc5591783986a8588d83646c11254caae742c40968dd9d7a5c" Oct 31 03:15:19.088793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.085186 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" event={"ID":"e58936d3-d580-4c84-9d7f-59502f7312b1","Type":"ContainerDied","Data":"9b18d767001f481470637ed58ed4eaa37f8e56a8f034fa30efe1a72b1e974bd8"} Oct 31 03:15:19.088793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.085336 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz" Oct 31 03:15:19.092238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.089555 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" event={"ID":"94bb5800-77c6-47a9-a7b4-ae82a1d2e074","Type":"ContainerDied","Data":"6d120befd12508f68954e29da6dfa24882b2b4894b127512fef90868ba62f294"} Oct 31 03:15:19.092908 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.092883 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" event={"ID":"159d54c2-29e1-4185-aed6-d7e839d50d4b","Type":"ContainerStarted","Data":"4bc2bc0c1f7c621db61209797b1d74083ad972baaf6059944bb9fdeae911071a"} Oct 31 03:15:19.093400 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.093373 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg" Oct 31 03:15:19.100714 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.100697 4456 scope.go:122] "RemoveContainer" containerID="34c57cd3a45f9ed9126225f5981f502c3b60eb9a331c88592e8b3c46f5804a6f" Oct 31 03:15:19.115638 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.115603 4456 scope.go:122] "RemoveContainer" containerID="596532fc271d79b939cd1ee338998fde4fcb86312d58c3e98cc5d3c0e15d64e3" Oct 31 03:15:19.117026 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.117006 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx"] Oct 31 03:15:19.121889 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.121864 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3fc2d0-29890275-4xzhx"] Oct 31 03:15:19.206929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.191956 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz"] Oct 31 03:15:19.206929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.206447 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d8daf-29890275-s9vmz"] Oct 31 03:15:19.213053 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.213022 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg"] Oct 31 03:15:19.217659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.217619 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebea4-29890275-k8mbg"] Oct 31 03:15:19.305008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.304982 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" Oct 31 03:15:19.411700 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.411671 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-kube-api-access-kscf8\" (UniqueName: \"kubernetes.io/projected/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-kube-api-access-kscf8\") pod \"895d18c5-e44e-46bb-bee5-f6bac0ec11ae\" (UID: \"895d18c5-e44e-46bb-bee5-f6bac0ec11ae\") " Oct 31 03:15:19.411865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.411832 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-maas-api-service-ca\") pod \"895d18c5-e44e-46bb-bee5-f6bac0ec11ae\" (UID: \"895d18c5-e44e-46bb-bee5-f6bac0ec11ae\") " Oct 31 03:15:19.412185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.412162 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-maas-api-service-ca" pod "895d18c5-e44e-46bb-bee5-f6bac0ec11ae" (UID: "895d18c5-e44e-46bb-bee5-f6bac0ec11ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:19.412401 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.412387 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-maas-api-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:19.446867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.446642 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-kube-api-access-kscf8" pod "895d18c5-e44e-46bb-bee5-f6bac0ec11ae" (UID: "895d18c5-e44e-46bb-bee5-f6bac0ec11ae"). InnerVolumeSpecName "kube-api-access-kscf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:19.513626 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:19.513600 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscf8\" (UniqueName: \"kubernetes.io/projected/895d18c5-e44e-46bb-bee5-f6bac0ec11ae-kube-api-access-kscf8\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:20.108107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.108012 4456 generic.go:356] "Generic (PLEG): container finished" podID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerID="4bc2bc0c1f7c621db61209797b1d74083ad972baaf6059944bb9fdeae911071a" exitCode=60 Oct 31 03:15:20.108107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.108064 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" event={"ID":"159d54c2-29e1-4185-aed6-d7e839d50d4b","Type":"ContainerDied","Data":"4bc2bc0c1f7c621db61209797b1d74083ad972baaf6059944bb9fdeae911071a"} Oct 31 03:15:20.108638 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.108121 4456 scope.go:122] "RemoveContainer" containerID="31198f27fe14cc57b9d5966ab2cc0b81f2536fdcdba912fe06b6ad70e397240a" Oct 31 03:15:20.108638 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.108613 4456 scope.go:122] "RemoveContainer" containerID="4bc2bc0c1f7c621db61209797b1d74083ad972baaf6059944bb9fdeae911071a" Oct 31 03:15:20.109177 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:20.109098 4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890275-m578d_odh-ai-gateway-infra(159d54c2-29e1-4185-aed6-d7e839d50d4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" Oct 31 03:15:20.111275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.111238 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" Oct 31 03:15:20.111275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.111260 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq" event={"ID":"895d18c5-e44e-46bb-bee5-f6bac0ec11ae","Type":"ContainerDied","Data":"5efb1958fb4e880198315e73a5955c7d9a92e7d8d91413d279f7c15025fa85d2"} Oct 31 03:15:20.126668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.126640 4456 scope.go:122] "RemoveContainer" containerID="a3e8cbd1c42a0a5d791c55f0ba7b9bac415e37b4fd6f62f66cf07ac57efd77f9" Oct 31 03:15:20.160372 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.160334 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq"] Oct 31 03:15:20.165150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.165120 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-56afbd-29890275-px2jq"] Oct 31 03:15:20.180373 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.180341 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d"] Oct 31 03:15:20.438443 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.438409 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" path="/var/lib/kubelet/pods/271d05c8-13c6-47ff-95d8-13d3653f9a3e/volumes" Oct 31 03:15:20.438928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.438912 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" path="/var/lib/kubelet/pods/895d18c5-e44e-46bb-bee5-f6bac0ec11ae/volumes" Oct 31 03:15:20.439384 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.439368 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" path="/var/lib/kubelet/pods/94bb5800-77c6-47a9-a7b4-ae82a1d2e074/volumes" Oct 31 03:15:20.440208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:20.440183 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" path="/var/lib/kubelet/pods/e58936d3-d580-4c84-9d7f-59502f7312b1/volumes" Oct 31 03:15:21.308428 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:21.308398 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" Oct 31 03:15:21.460508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:21.460265 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159d54c2-29e1-4185-aed6-d7e839d50d4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159d54c2-29e1-4185-aed6-d7e839d50d4b-maas-api-service-ca\") pod \"159d54c2-29e1-4185-aed6-d7e839d50d4b\" (UID: \"159d54c2-29e1-4185-aed6-d7e839d50d4b\") " Oct 31 03:15:21.460508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:21.460450 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159d54c2-29e1-4185-aed6-d7e839d50d4b-kube-api-access-5s7gc\" (UniqueName: \"kubernetes.io/projected/159d54c2-29e1-4185-aed6-d7e839d50d4b-kube-api-access-5s7gc\") pod \"159d54c2-29e1-4185-aed6-d7e839d50d4b\" (UID: \"159d54c2-29e1-4185-aed6-d7e839d50d4b\") " Oct 31 03:15:21.462148 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:21.462118 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159d54c2-29e1-4185-aed6-d7e839d50d4b-maas-api-service-ca" pod "159d54c2-29e1-4185-aed6-d7e839d50d4b" (UID: "159d54c2-29e1-4185-aed6-d7e839d50d4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:21.462934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:21.462909 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159d54c2-29e1-4185-aed6-d7e839d50d4b-maas-api-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:21.473971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:21.473938 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159d54c2-29e1-4185-aed6-d7e839d50d4b-kube-api-access-5s7gc" pod "159d54c2-29e1-4185-aed6-d7e839d50d4b" (UID: "159d54c2-29e1-4185-aed6-d7e839d50d4b"). InnerVolumeSpecName "kube-api-access-5s7gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:21.564726 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:21.564682 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s7gc\" (UniqueName: \"kubernetes.io/projected/159d54c2-29e1-4185-aed6-d7e839d50d4b-kube-api-access-5s7gc\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:22.138484 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:22.138450 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" Oct 31 03:15:22.138652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:22.138451 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d" event={"ID":"159d54c2-29e1-4185-aed6-d7e839d50d4b","Type":"ContainerDied","Data":"51b478e8dad1bbddc9d9019f99863cfeb40cde94e21fbb3dc5c9e9d56d76afed"} Oct 31 03:15:22.138652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:22.138581 4456 scope.go:122] "RemoveContainer" containerID="4bc2bc0c1f7c621db61209797b1d74083ad972baaf6059944bb9fdeae911071a" Oct 31 03:15:22.173573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:22.173540 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d"] Oct 31 03:15:22.177837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:22.177810 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-m578d"] Oct 31 03:15:22.437684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:22.437641 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" path="/var/lib/kubelet/pods/159d54c2-29e1-4185-aed6-d7e839d50d4b/volumes" Oct 31 03:15:30.634389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.634352 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b"] Oct 31 03:15:30.635099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635080 4456 cpu_manager.go:414] "containerMap: removing container" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635101 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635115 4456 cpu_manager.go:414] "containerMap: removing container" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635123 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635132 4456 cpu_manager.go:414] "containerMap: removing container" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635137 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635148 4456 cpu_manager.go:414] "containerMap: removing container" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635153 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635159 4456 cpu_manager.go:414] "containerMap: removing container" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635164 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635173 4456 cpu_manager.go:414] "containerMap: removing container" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635177 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635185 4456 cpu_manager.go:414] "containerMap: removing container" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635189 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635197 4456 cpu_manager.go:414] "containerMap: removing container" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerName="cleanup" Oct 31 03:15:30.635194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635201 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635209 4456 cpu_manager.go:414] "containerMap: removing container" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635214 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635234 4456 cpu_manager.go:414] "containerMap: removing container" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635238 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635243 4456 cpu_manager.go:414] "containerMap: removing container" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635248 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635263 4456 cpu_manager.go:414] "containerMap: removing container" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635267 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635278 4456 cpu_manager.go:414] "containerMap: removing container" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635283 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635406 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635413 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635419 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635424 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635431 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635437 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635444 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635451 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635455 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="159d54c2-29e1-4185-aed6-d7e839d50d4b" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635462 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635470 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerName="cleanup" Oct 31 03:15:30.635655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.635476 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerName="cleanup" Oct 31 03:15:30.640812 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.640795 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:15:30.648145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.647493 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-2206e7\"" Oct 31 03:15:30.648145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.647864 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-2206e7-dockercfg-kgxtq\"" Oct 31 03:15:30.660623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.660595 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b"] Oct 31 03:15:30.798764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.789929 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdvgm\" (UniqueName: \"kubernetes.io/projected/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-kube-api-access-mdvgm\") pod \"payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b\" (UID: \"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:15:30.798764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.790147 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b\" (UID: \"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:15:30.911107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.899222 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b\" (UID: \"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:15:30.911107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.899552 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdvgm\" (UniqueName: \"kubernetes.io/projected/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-kube-api-access-mdvgm\") pod \"payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b\" (UID: \"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:15:30.911107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.907880 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b\" (UID: \"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:15:30.946343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.946306 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdvgm\" (UniqueName: \"kubernetes.io/projected/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-kube-api-access-mdvgm\") pod \"payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b\" (UID: \"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:15:30.962383 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:30.962350 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:15:31.322943 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:15:31.322885 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6986cbad_3cdf_4333_b1ab_d49bc8bc3eea.slice/crio-61666d8b775a6b15acb26d0ff9e2c500d5ef777c71cc1aff4e734e85fd6e267f WatchSource:0}: Error finding container 61666d8b775a6b15acb26d0ff9e2c500d5ef777c71cc1aff4e734e85fd6e267f: Status 404 returned error can't find the container with id 61666d8b775a6b15acb26d0ff9e2c500d5ef777c71cc1aff4e734e85fd6e267f Oct 31 03:15:31.323276 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.323099 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b"] Oct 31 03:15:31.784817 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.784776 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm"] Oct 31 03:15:31.786506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.786461 4456 cpu_manager.go:414] "containerMap: removing container" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerName="cleanup" Oct 31 03:15:31.786722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.786707 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bb5800-77c6-47a9-a7b4-ae82a1d2e074" containerName="cleanup" Oct 31 03:15:31.786915 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.786902 4456 cpu_manager.go:414] "containerMap: removing container" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerName="cleanup" Oct 31 03:15:31.787018 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.787006 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerName="cleanup" Oct 31 03:15:31.787343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.787328 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="271d05c8-13c6-47ff-95d8-13d3653f9a3e" containerName="cleanup" Oct 31 03:15:31.787458 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.787437 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58936d3-d580-4c84-9d7f-59502f7312b1" containerName="cleanup" Oct 31 03:15:31.787583 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.787464 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="895d18c5-e44e-46bb-bee5-f6bac0ec11ae" containerName="cleanup" Oct 31 03:15:31.792277 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.792254 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:15:31.812101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.811980 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm"] Oct 31 03:15:31.835776 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.835439 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c40da77-42b0-4315-bf18-d38ecda7f5ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm\" (UID: \"7c40da77-42b0-4315-bf18-d38ecda7f5ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:15:31.835776 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.835504 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptbnt\" (UniqueName: \"kubernetes.io/projected/7c40da77-42b0-4315-bf18-d38ecda7f5ce-kube-api-access-ptbnt\") pod \"payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm\" (UID: \"7c40da77-42b0-4315-bf18-d38ecda7f5ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:15:31.869731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.869614 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-750c955f\"" Oct 31 03:15:31.937018 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.936975 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c40da77-42b0-4315-bf18-d38ecda7f5ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm\" (UID: \"7c40da77-42b0-4315-bf18-d38ecda7f5ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:15:31.937260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.937098 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptbnt\" (UniqueName: \"kubernetes.io/projected/7c40da77-42b0-4315-bf18-d38ecda7f5ce-kube-api-access-ptbnt\") pod \"payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm\" (UID: \"7c40da77-42b0-4315-bf18-d38ecda7f5ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:15:31.937943 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.937902 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c40da77-42b0-4315-bf18-d38ecda7f5ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm\" (UID: \"7c40da77-42b0-4315-bf18-d38ecda7f5ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:15:31.995066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:31.993953 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptbnt\" (UniqueName: \"kubernetes.io/projected/7c40da77-42b0-4315-bf18-d38ecda7f5ce-kube-api-access-ptbnt\") pod \"payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm\" (UID: \"7c40da77-42b0-4315-bf18-d38ecda7f5ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:15:32.154775 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:32.153300 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:15:32.222018 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:15:32.221899 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:15:32.345258 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:32.345217 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" event={"ID":"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea","Type":"ContainerStarted","Data":"c6ac8d53a441ea9dcdaf99a529d5fe577a01748dcee0a5dd267a5ca105a4f37f"} Oct 31 03:15:32.345481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:32.345288 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" event={"ID":"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea","Type":"ContainerStarted","Data":"61666d8b775a6b15acb26d0ff9e2c500d5ef777c71cc1aff4e734e85fd6e267f"} Oct 31 03:15:32.345481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:32.345347 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:15:32.375626 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:32.375558 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" podStartSLOduration=2.375538982 podStartE2EDuration="2.375538982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:32.373221587 +0000 UTC m=+1214.053747714" watchObservedRunningTime="2026-10-31 03:15:32.375538982 +0000 UTC m=+1214.056065109" Oct 31 03:15:32.518926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:32.518881 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm"] Oct 31 03:15:33.359096 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:33.359007 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" event={"ID":"7c40da77-42b0-4315-bf18-d38ecda7f5ce","Type":"ContainerStarted","Data":"a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112"} Oct 31 03:15:33.359600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:33.359117 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" event={"ID":"7c40da77-42b0-4315-bf18-d38ecda7f5ce","Type":"ContainerStarted","Data":"90a2e40ffb2a31889204dcfe871ac983c042a316a70d210c9a61f3a0171aa258"} Oct 31 03:15:33.381743 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:33.381689 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" podStartSLOduration=2.381671409 podStartE2EDuration="2.381671409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:33.380802663 +0000 UTC m=+1215.061328792" watchObservedRunningTime="2026-10-31 03:15:33.381671409 +0000 UTC m=+1215.062197538" Oct 31 03:15:34.371504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:34.371449 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:15:41.669542 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.669506 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs"] Oct 31 03:15:41.680267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.680241 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs"] Oct 31 03:15:41.680488 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.680366 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:15:41.686452 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.686428 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2ad1f0\"" Oct 31 03:15:41.688759 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.688731 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2ad1f0-dockercfg-54dz7\"" Oct 31 03:15:41.876061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.875438 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fca5e6e-5a1b-45f8-838e-1fa234468a78-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs\" (UID: \"7fca5e6e-5a1b-45f8-838e-1fa234468a78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:15:41.876061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.875501 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scdm\" (UniqueName: \"kubernetes.io/projected/7fca5e6e-5a1b-45f8-838e-1fa234468a78-kube-api-access-5scdm\") pod \"payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs\" (UID: \"7fca5e6e-5a1b-45f8-838e-1fa234468a78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:15:41.977319 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.977233 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fca5e6e-5a1b-45f8-838e-1fa234468a78-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs\" (UID: \"7fca5e6e-5a1b-45f8-838e-1fa234468a78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:15:41.977319 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.977278 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scdm\" (UniqueName: \"kubernetes.io/projected/7fca5e6e-5a1b-45f8-838e-1fa234468a78-kube-api-access-5scdm\") pod \"payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs\" (UID: \"7fca5e6e-5a1b-45f8-838e-1fa234468a78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:15:41.977944 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.977922 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fca5e6e-5a1b-45f8-838e-1fa234468a78-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs\" (UID: \"7fca5e6e-5a1b-45f8-838e-1fa234468a78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:15:41.990966 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:41.990931 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scdm\" (UniqueName: \"kubernetes.io/projected/7fca5e6e-5a1b-45f8-838e-1fa234468a78-kube-api-access-5scdm\") pod \"payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs\" (UID: \"7fca5e6e-5a1b-45f8-838e-1fa234468a78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:15:42.008310 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:42.007646 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:15:42.377508 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:15:42.377479 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fca5e6e_5a1b_45f8_838e_1fa234468a78.slice/crio-5b8a540db4fc5b29eb16d980c5df78750ab2d5aa56e153a8a2d8bcf95ab0c723 WatchSource:0}: Error finding container 5b8a540db4fc5b29eb16d980c5df78750ab2d5aa56e153a8a2d8bcf95ab0c723: Status 404 returned error can't find the container with id 5b8a540db4fc5b29eb16d980c5df78750ab2d5aa56e153a8a2d8bcf95ab0c723 Oct 31 03:15:42.378696 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:42.378665 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs"] Oct 31 03:15:42.532762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:42.532726 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" event={"ID":"7fca5e6e-5a1b-45f8-838e-1fa234468a78","Type":"ContainerStarted","Data":"ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8"} Oct 31 03:15:42.532944 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:42.532776 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" event={"ID":"7fca5e6e-5a1b-45f8-838e-1fa234468a78","Type":"ContainerStarted","Data":"5b8a540db4fc5b29eb16d980c5df78750ab2d5aa56e153a8a2d8bcf95ab0c723"} Oct 31 03:15:42.533011 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:42.532955 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:15:42.562395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:42.562342 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" podStartSLOduration=1.562329582 podStartE2EDuration="1.562329582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:42.556959349 +0000 UTC m=+1224.237485475" watchObservedRunningTime="2026-10-31 03:15:42.562329582 +0000 UTC m=+1224.242855708" Oct 31 03:15:43.363860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:43.363829 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:15:45.390441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:45.390407 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:15:50.069167 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.069124 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt"] Oct 31 03:15:50.075750 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.075723 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.105426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.105399 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt"] Oct 31 03:15:50.164254 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.164220 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f1bfbe55-openshift-default-dockercfg-djzcs\"" Oct 31 03:15:50.187448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.187407 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.187620 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.187522 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.187620 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.187569 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470600d7-dfd0-4741-9cdf-a840d77d5beb-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.187712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.187659 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.187766 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.187712 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.187879 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.187858 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptl2k\" (UniqueName: \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-kube-api-access-ptl2k\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.187950 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.187920 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.188012 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.187988 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.188142 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.188125 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.289830 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.289797 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.289851 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.289886 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470600d7-dfd0-4741-9cdf-a840d77d5beb-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.289931 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.289964 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.290098 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptl2k\" (UniqueName: \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-kube-api-access-ptl2k\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.290146 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.290177 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.290242 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.290332 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290617 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.290598 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.290785 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.290889 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.290826 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.291608 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.291582 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470600d7-dfd0-4741-9cdf-a840d77d5beb-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.292695 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.292675 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.292881 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.292862 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.308186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.308164 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.308442 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.308420 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptl2k\" (UniqueName: \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-kube-api-access-ptl2k\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.405247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.405220 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:50.714363 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:15:50.714321 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470600d7_dfd0_4741_9cdf_a840d77d5beb.slice/crio-12d0842043ebdd5db19ff51a2a75f79aac54867a4cb5b3a6ac0e3eaa9a1d222d WatchSource:0}: Error finding container 12d0842043ebdd5db19ff51a2a75f79aac54867a4cb5b3a6ac0e3eaa9a1d222d: Status 404 returned error can't find the container with id 12d0842043ebdd5db19ff51a2a75f79aac54867a4cb5b3a6ac0e3eaa9a1d222d Oct 31 03:15:50.716413 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.716388 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt"] Oct 31 03:15:50.717957 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.717918 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:15:50.718071 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.717995 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:15:50.718071 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:50.718030 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:15:51.632861 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:51.632823 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" event={"ID":"470600d7-dfd0-4741-9cdf-a840d77d5beb","Type":"ContainerStarted","Data":"f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1"} Oct 31 03:15:51.633274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:51.632906 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" event={"ID":"470600d7-dfd0-4741-9cdf-a840d77d5beb","Type":"ContainerStarted","Data":"12d0842043ebdd5db19ff51a2a75f79aac54867a4cb5b3a6ac0e3eaa9a1d222d"} Oct 31 03:15:51.684761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:51.684704 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" podStartSLOduration=1.6846850880000002 podStartE2EDuration="1.684685088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:51.669884393 +0000 UTC m=+1233.350410546" watchObservedRunningTime="2026-10-31 03:15:51.684685088 +0000 UTC m=+1233.365211217" Oct 31 03:15:52.407266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:52.407182 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:52.409689 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:52.409653 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:15:52.409836 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:52.409707 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:15:53.405576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:53.405542 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:15:53.405994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:53.405596 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:15:53.544473 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:53.544433 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:15:54.406663 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:54.406614 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:15:54.407347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:54.407308 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:15:55.409427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:55.409389 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:15:55.410064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:55.410017 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:15:56.418175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:56.418131 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:15:56.418780 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:56.418743 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:15:57.406585 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:57.406551 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:15:57.406745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:57.406609 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:15:58.411526 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:58.411490 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:58.412339 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:58.412312 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:15:58.413913 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:15:58.413887 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:16:02.182107 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:16:02.182018 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:16:07.185073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.185020 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4"] Oct 31 03:16:07.191950 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.191923 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:16:07.199708 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.199673 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-0a531d\"" Oct 31 03:16:07.200983 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.200962 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4"] Oct 31 03:16:07.221969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.221943 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg"] Oct 31 03:16:07.227299 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.227276 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:16:07.231254 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.231227 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-0a531d-dockercfg-mdd4m\"" Oct 31 03:16:07.238440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.238411 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg"] Oct 31 03:16:07.314213 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.314107 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f819a08-2680-493d-80a4-d84b281d78e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4\" (UID: \"1f819a08-2680-493d-80a4-d84b281d78e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:16:07.314408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.314263 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7knx\" (UniqueName: \"kubernetes.io/projected/1f819a08-2680-493d-80a4-d84b281d78e4-kube-api-access-s7knx\") pod \"payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4\" (UID: \"1f819a08-2680-493d-80a4-d84b281d78e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:16:07.314408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.314385 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ebb1b0-8737-4275-8c09-5d5152251c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg\" (UID: \"72ebb1b0-8737-4275-8c09-5d5152251c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:16:07.314581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.314547 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9qv\" (UniqueName: \"kubernetes.io/projected/72ebb1b0-8737-4275-8c09-5d5152251c81-kube-api-access-nm9qv\") pod \"payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg\" (UID: \"72ebb1b0-8737-4275-8c09-5d5152251c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:16:07.416060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.416001 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9qv\" (UniqueName: \"kubernetes.io/projected/72ebb1b0-8737-4275-8c09-5d5152251c81-kube-api-access-nm9qv\") pod \"payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg\" (UID: \"72ebb1b0-8737-4275-8c09-5d5152251c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:16:07.417382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.417345 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f819a08-2680-493d-80a4-d84b281d78e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4\" (UID: \"1f819a08-2680-493d-80a4-d84b281d78e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:16:07.418504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.418396 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f819a08-2680-493d-80a4-d84b281d78e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4\" (UID: \"1f819a08-2680-493d-80a4-d84b281d78e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:16:07.434017 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.433990 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7knx\" (UniqueName: \"kubernetes.io/projected/1f819a08-2680-493d-80a4-d84b281d78e4-kube-api-access-s7knx\") pod \"payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4\" (UID: \"1f819a08-2680-493d-80a4-d84b281d78e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:16:07.434739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.434716 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ebb1b0-8737-4275-8c09-5d5152251c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg\" (UID: \"72ebb1b0-8737-4275-8c09-5d5152251c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:16:07.441114 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.435795 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ebb1b0-8737-4275-8c09-5d5152251c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg\" (UID: \"72ebb1b0-8737-4275-8c09-5d5152251c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:16:07.441304 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.436382 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9qv\" (UniqueName: \"kubernetes.io/projected/72ebb1b0-8737-4275-8c09-5d5152251c81-kube-api-access-nm9qv\") pod \"payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg\" (UID: \"72ebb1b0-8737-4275-8c09-5d5152251c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:16:07.459059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.454871 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7knx\" (UniqueName: \"kubernetes.io/projected/1f819a08-2680-493d-80a4-d84b281d78e4-kube-api-access-s7knx\") pod \"payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4\" (UID: \"1f819a08-2680-493d-80a4-d84b281d78e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:16:07.528061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.527707 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:16:07.548607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.548574 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:16:07.915971 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:16:07.915940 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ebb1b0_8737_4275_8c09_5d5152251c81.slice/crio-858079f5c21aed2be3c8e42534b53065adaf7862cf95a93df44541a7b713f00e WatchSource:0}: Error finding container 858079f5c21aed2be3c8e42534b53065adaf7862cf95a93df44541a7b713f00e: Status 404 returned error can't find the container with id 858079f5c21aed2be3c8e42534b53065adaf7862cf95a93df44541a7b713f00e Oct 31 03:16:07.916308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.916281 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4"] Oct 31 03:16:07.924082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:07.924055 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg"] Oct 31 03:16:08.920691 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:08.920643 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" event={"ID":"72ebb1b0-8737-4275-8c09-5d5152251c81","Type":"ContainerStarted","Data":"8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21"} Oct 31 03:16:08.921186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:08.920713 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" event={"ID":"72ebb1b0-8737-4275-8c09-5d5152251c81","Type":"ContainerStarted","Data":"858079f5c21aed2be3c8e42534b53065adaf7862cf95a93df44541a7b713f00e"} Oct 31 03:16:08.921186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:08.920744 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:16:08.922600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:08.922571 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" event={"ID":"1f819a08-2680-493d-80a4-d84b281d78e4","Type":"ContainerStarted","Data":"275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c"} Oct 31 03:16:08.922735 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:08.922628 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:16:08.922735 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:08.922653 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" event={"ID":"1f819a08-2680-493d-80a4-d84b281d78e4","Type":"ContainerStarted","Data":"94cff9bd7d5a784951db96207e55306149ace1d8433bbc35984d8983c616bd7d"} Oct 31 03:16:08.948551 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:08.948500 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" podStartSLOduration=1.948485889 podStartE2EDuration="1.948485889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:08.947932835 +0000 UTC m=+1250.628458962" watchObservedRunningTime="2026-10-31 03:16:08.948485889 +0000 UTC m=+1250.629012028" Oct 31 03:16:08.971756 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:08.971702 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" podStartSLOduration=1.97168806 podStartE2EDuration="1.97168806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:08.968957167 +0000 UTC m=+1250.649483293" watchObservedRunningTime="2026-10-31 03:16:08.97168806 +0000 UTC m=+1250.652214187" Oct 31 03:16:10.443483 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.443115 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5"] Oct 31 03:16:10.448466 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.448442 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5"] Oct 31 03:16:10.448611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.448535 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:10.455785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.455761 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:16:10.456117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.456075 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:16:10.496627 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.496586 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83a30d1-dtfb5\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:10.496802 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.496760 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjjvn\" (UniqueName: \"kubernetes.io/projected/175a71f2-f2bd-4139-a641-1e7d357b04e0-kube-api-access-wjjvn\") pod \"maas-api-revoke-sub-a83a30d1-dtfb5\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:10.497115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.497083 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83a30d1-dtfb5\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:10.598546 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.598504 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjvn\" (UniqueName: \"kubernetes.io/projected/175a71f2-f2bd-4139-a641-1e7d357b04e0-kube-api-access-wjjvn\") pod \"maas-api-revoke-sub-a83a30d1-dtfb5\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:10.598752 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.598678 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83a30d1-dtfb5\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:10.598865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.598849 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83a30d1-dtfb5\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:10.599485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.599458 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83a30d1-dtfb5\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:10.601322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.601204 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83a30d1-dtfb5\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:10.611955 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.611925 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjjvn\" (UniqueName: \"kubernetes.io/projected/175a71f2-f2bd-4139-a641-1e7d357b04e0-kube-api-access-wjjvn\") pod \"maas-api-revoke-sub-a83a30d1-dtfb5\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:10.787485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:10.787388 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:11.100378 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:11.100349 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5"] Oct 31 03:16:11.100501 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:16:11.100347 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175a71f2_f2bd_4139_a641_1e7d357b04e0.slice/crio-60ca7a69ded6334a0fca9a3f62ec34b582fbe9aea23ffdd5369929328b20408d WatchSource:0}: Error finding container 60ca7a69ded6334a0fca9a3f62ec34b582fbe9aea23ffdd5369929328b20408d: Status 404 returned error can't find the container with id 60ca7a69ded6334a0fca9a3f62ec34b582fbe9aea23ffdd5369929328b20408d Oct 31 03:16:11.992720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:11.992673 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" event={"ID":"175a71f2-f2bd-4139-a641-1e7d357b04e0","Type":"ContainerStarted","Data":"c1e5b833c405fae66a87598676258667032d924bc0a2c8cac82ddacd3bbf95e2"} Oct 31 03:16:11.993204 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:11.992743 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" event={"ID":"175a71f2-f2bd-4139-a641-1e7d357b04e0","Type":"ContainerStarted","Data":"60ca7a69ded6334a0fca9a3f62ec34b582fbe9aea23ffdd5369929328b20408d"} Oct 31 03:16:12.015577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:12.015524 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" podStartSLOduration=2.015508293 podStartE2EDuration="2.015508293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:12.013989034 +0000 UTC m=+1253.694515162" watchObservedRunningTime="2026-10-31 03:16:12.015508293 +0000 UTC m=+1253.696034419" Oct 31 03:16:13.007147 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:13.007096 4456 generic.go:356] "Generic (PLEG): container finished" podID="175a71f2-f2bd-4139-a641-1e7d357b04e0" containerID="c1e5b833c405fae66a87598676258667032d924bc0a2c8cac82ddacd3bbf95e2" exitCode=0 Oct 31 03:16:13.007566 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:13.007209 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" event={"ID":"175a71f2-f2bd-4139-a641-1e7d357b04e0","Type":"ContainerDied","Data":"c1e5b833c405fae66a87598676258667032d924bc0a2c8cac82ddacd3bbf95e2"} Oct 31 03:16:14.284700 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:14.284614 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:14.396070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:14.394446 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175a71f2-f2bd-4139-a641-1e7d357b04e0-kube-api-access-wjjvn\" (UniqueName: \"kubernetes.io/projected/175a71f2-f2bd-4139-a641-1e7d357b04e0-kube-api-access-wjjvn\") pod \"175a71f2-f2bd-4139-a641-1e7d357b04e0\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " Oct 31 03:16:14.396070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:14.394585 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-openshift-ca\") pod \"175a71f2-f2bd-4139-a641-1e7d357b04e0\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " Oct 31 03:16:14.396070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:14.394637 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-xks-ca\") pod \"175a71f2-f2bd-4139-a641-1e7d357b04e0\" (UID: \"175a71f2-f2bd-4139-a641-1e7d357b04e0\") " Oct 31 03:16:14.414377 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:14.413488 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-openshift-ca" pod "175a71f2-f2bd-4139-a641-1e7d357b04e0" (UID: "175a71f2-f2bd-4139-a641-1e7d357b04e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:14.433222 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:14.433160 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-xks-ca" pod "175a71f2-f2bd-4139-a641-1e7d357b04e0" (UID: "175a71f2-f2bd-4139-a641-1e7d357b04e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:14.438706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:14.438651 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175a71f2-f2bd-4139-a641-1e7d357b04e0-kube-api-access-wjjvn" pod "175a71f2-f2bd-4139-a641-1e7d357b04e0" (UID: "175a71f2-f2bd-4139-a641-1e7d357b04e0"). InnerVolumeSpecName "kube-api-access-wjjvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:16:14.496975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:14.496929 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjjvn\" (UniqueName: \"kubernetes.io/projected/175a71f2-f2bd-4139-a641-1e7d357b04e0-kube-api-access-wjjvn\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:14.496975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:14.496960 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:14.496975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:14.496972 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175a71f2-f2bd-4139-a641-1e7d357b04e0-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:15.026182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:15.026141 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" event={"ID":"175a71f2-f2bd-4139-a641-1e7d357b04e0","Type":"ContainerDied","Data":"60ca7a69ded6334a0fca9a3f62ec34b582fbe9aea23ffdd5369929328b20408d"} Oct 31 03:16:15.026182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:15.026171 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5" Oct 31 03:16:15.026428 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:15.026192 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60ca7a69ded6334a0fca9a3f62ec34b582fbe9aea23ffdd5369929328b20408d" Oct 31 03:16:17.654870 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.654833 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx"] Oct 31 03:16:17.656356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.656328 4456 cpu_manager.go:414] "containerMap: removing container" podUID="175a71f2-f2bd-4139-a641-1e7d357b04e0" containerName="revoke-keys" Oct 31 03:16:17.656356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.656353 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175a71f2-f2bd-4139-a641-1e7d357b04e0" containerName="revoke-keys" Oct 31 03:16:17.656511 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.656502 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="175a71f2-f2bd-4139-a641-1e7d357b04e0" containerName="revoke-keys" Oct 31 03:16:17.660739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.660718 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:16:17.668493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.668445 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-780abd-dockercfg-6ml9j\"" Oct 31 03:16:17.668641 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.668511 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-780abd\"" Oct 31 03:16:17.673320 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.673293 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx"] Oct 31 03:16:17.774418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.774384 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490d58df-8d11-4c30-9d01-55f5540c490e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx\" (UID: \"490d58df-8d11-4c30-9d01-55f5540c490e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:16:17.774810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.774779 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4klvp\" (UniqueName: \"kubernetes.io/projected/490d58df-8d11-4c30-9d01-55f5540c490e-kube-api-access-4klvp\") pod \"payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx\" (UID: \"490d58df-8d11-4c30-9d01-55f5540c490e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:16:17.881055 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.878013 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4klvp\" (UniqueName: \"kubernetes.io/projected/490d58df-8d11-4c30-9d01-55f5540c490e-kube-api-access-4klvp\") pod \"payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx\" (UID: \"490d58df-8d11-4c30-9d01-55f5540c490e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:16:17.881055 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.878415 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490d58df-8d11-4c30-9d01-55f5540c490e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx\" (UID: \"490d58df-8d11-4c30-9d01-55f5540c490e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:16:17.881055 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.879291 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490d58df-8d11-4c30-9d01-55f5540c490e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx\" (UID: \"490d58df-8d11-4c30-9d01-55f5540c490e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:16:17.903762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.903737 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4klvp\" (UniqueName: \"kubernetes.io/projected/490d58df-8d11-4c30-9d01-55f5540c490e-kube-api-access-4klvp\") pod \"payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx\" (UID: \"490d58df-8d11-4c30-9d01-55f5540c490e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:16:17.988692 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:17.988587 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:16:18.292773 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:16:18.292728 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490d58df_8d11_4c30_9d01_55f5540c490e.slice/crio-872b88877b117bcd4d554d760c6d04476e3810f4c033f4cad49fd67bc23f855d WatchSource:0}: Error finding container 872b88877b117bcd4d554d760c6d04476e3810f4c033f4cad49fd67bc23f855d: Status 404 returned error can't find the container with id 872b88877b117bcd4d554d760c6d04476e3810f4c033f4cad49fd67bc23f855d Oct 31 03:16:18.296184 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:18.294961 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx"] Oct 31 03:16:19.185088 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:19.175324 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" event={"ID":"490d58df-8d11-4c30-9d01-55f5540c490e","Type":"ContainerStarted","Data":"e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30"} Oct 31 03:16:19.185088 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:19.175442 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" event={"ID":"490d58df-8d11-4c30-9d01-55f5540c490e","Type":"ContainerStarted","Data":"872b88877b117bcd4d554d760c6d04476e3810f4c033f4cad49fd67bc23f855d"} Oct 31 03:16:19.185088 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:19.176631 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:16:19.259057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:19.238599 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" podStartSLOduration=2.238580594 podStartE2EDuration="2.238580594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:19.234119492 +0000 UTC m=+1260.914645620" watchObservedRunningTime="2026-10-31 03:16:19.238580594 +0000 UTC m=+1260.919106722" Oct 31 03:16:19.933679 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:19.933646 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:16:19.933849 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:19.933727 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:16:25.500315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.500277 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq"] Oct 31 03:16:25.509939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.507667 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.521310 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.519132 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-2206e7-openshift-default-dockercfg-t49h2\"" Oct 31 03:16:25.521310 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.520964 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq"] Oct 31 03:16:25.630052 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.630011 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.630224 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.630115 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.630224 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.630159 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.630224 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.630190 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6bf6\" (UniqueName: \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-kube-api-access-x6bf6\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.630366 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.630344 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2dcbdd-092e-492e-982b-0b7c070503f4-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.630413 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.630378 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.630594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.630575 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.630670 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.630640 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.630727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.630694 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.732522 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.732580 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.732625 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.732670 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6bf6\" (UniqueName: \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-kube-api-access-x6bf6\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.732733 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2dcbdd-092e-492e-982b-0b7c070503f4-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.732763 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.732951 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.733011 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.733082 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.733438 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.734059 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2dcbdd-092e-492e-982b-0b7c070503f4-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734753 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.734314 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.734753 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.734532 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.735382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.735360 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.746718 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.746688 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.771944 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.769368 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.780471 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.780431 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.790549 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.788687 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6bf6\" (UniqueName: \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-kube-api-access-x6bf6\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:25.849499 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:25.849149 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:26.212260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:26.212223 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq"] Oct 31 03:16:26.216940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:26.216896 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:26.217085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:26.216995 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:26.217085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:26.217055 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:26.276500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:26.276457 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" event={"ID":"be2dcbdd-092e-492e-982b-0b7c070503f4","Type":"ContainerStarted","Data":"b1ebc320b43643173229bb9f249d8416893eecb1b4ada641501c81793e2b8fc9"} Oct 31 03:16:27.324802 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:27.324754 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" event={"ID":"be2dcbdd-092e-492e-982b-0b7c070503f4","Type":"ContainerStarted","Data":"6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab"} Oct 31 03:16:27.363988 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:27.363919 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" podStartSLOduration=2.363890825 podStartE2EDuration="2.363890825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:27.356155538 +0000 UTC m=+1269.036681667" watchObservedRunningTime="2026-10-31 03:16:27.363890825 +0000 UTC m=+1269.044416954" Oct 31 03:16:27.866011 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:27.865978 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:27.867207 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:27.867174 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:16:27.867433 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:27.867402 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:16:28.846065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:28.830096 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6980-7bae42"] Oct 31 03:16:28.861323 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:28.861287 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:28.878065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:28.864995 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6980-7bae42"] Oct 31 03:16:29.270229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.250396 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bshq7\"" Oct 31 03:16:29.270229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.251230 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:16:29.270229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.251280 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:16:29.270229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.260428 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c97lc\" (UniqueName: \"kubernetes.io/projected/b521178c-351d-4c86-852c-8c92ef0fa0a0-kube-api-access-c97lc\") pod \"test-curl-6980-7bae42\" (UID: \"b521178c-351d-4c86-852c-8c92ef0fa0a0\") " pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:29.270229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.264946 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:29.303062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.293155 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:29.375115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.363228 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c97lc\" (UniqueName: \"kubernetes.io/projected/b521178c-351d-4c86-852c-8c92ef0fa0a0-kube-api-access-c97lc\") pod \"test-curl-6980-7bae42\" (UID: \"b521178c-351d-4c86-852c-8c92ef0fa0a0\") " pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:29.411529 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.411488 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:29.596781 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.596642 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c97lc\" (UniqueName: \"kubernetes.io/projected/b521178c-351d-4c86-852c-8c92ef0fa0a0-kube-api-access-c97lc\") pod \"test-curl-6980-7bae42\" (UID: \"b521178c-351d-4c86-852c-8c92ef0fa0a0\") " pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:29.707301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.704825 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:29.874114 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.861187 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:16:29.874114 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.861255 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:16:29.874114 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:29.861639 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:30.518109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:30.512030 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 03:16:30.571136 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:30.562997 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:30.627061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:30.614356 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:16:30.828168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:30.828117 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:16:30.980665 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:30.980634 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:16:30.981294 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:30.981276 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:16:30.981469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:30.981456 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:16:31.002063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:31.002019 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:16:31.030284 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:31.030247 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:16:31.030531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:31.030506 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:16:31.132974 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:31.131466 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:16:31.194918 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:31.189700 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6980-7bae42"] Oct 31 03:16:31.204936 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:31.204910 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:16:31.838382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:31.838346 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6980-7bae42" event={"ID":"b521178c-351d-4c86-852c-8c92ef0fa0a0","Type":"ContainerStarted","Data":"ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff"} Oct 31 03:16:31.838597 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:31.838580 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6980-7bae42" event={"ID":"b521178c-351d-4c86-852c-8c92ef0fa0a0","Type":"ContainerStarted","Data":"bd6d9f55fa70c14ccf3a03bbd8c7d71262b4d7cda8254a783d159c2da1097a4e"} Oct 31 03:16:31.876169 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:31.876128 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:16:31.876438 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:31.876405 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:16:32.232731 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:16:32.232679 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:16:32.861060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:32.860764 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:16:32.861060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:32.860819 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:16:33.856095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:33.856060 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:33.856694 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:33.856532 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:33.857463 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:33.857430 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:16:33.891408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:33.891347 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6980-7bae42" podStartSLOduration=5.891329202 podStartE2EDuration="5.891329202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:31.87134195 +0000 UTC m=+1273.551868078" watchObservedRunningTime="2026-10-31 03:16:33.891329202 +0000 UTC m=+1275.571855329" Oct 31 03:16:35.143079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.143004 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g"] Oct 31 03:16:35.148234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.148211 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.194156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.192740 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5"] Oct 31 03:16:35.214587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.214561 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.225448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.214446 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g"] Oct 31 03:16:35.225448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.221357 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5"] Oct 31 03:16:35.234556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.233423 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.234556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.233538 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.234556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.233589 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.234556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.233618 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.234556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.233634 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gv72\" (UniqueName: \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-kube-api-access-6gv72\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.234556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.233674 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.234556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.233714 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.234556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.233766 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.234556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.233829 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.335563 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.335525 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.335755 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.335600 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.335755 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.335648 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.335755 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.335664 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.335755 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.335689 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.335755 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.335709 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.336113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.335852 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsksk\" (UniqueName: \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-kube-api-access-bsksk\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.336113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.335916 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.336113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336001 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.336113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336007 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.336113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336019 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.336356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336165 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.336356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336206 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.336356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336316 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.336356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336344 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.336565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336384 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.336565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336403 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9354b4b-c613-48a1-aebf-f2eed6be09da-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.336565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336508 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.336565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336540 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gv72\" (UniqueName: \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-kube-api-access-6gv72\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.336565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336544 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.336804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336572 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.336804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336677 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.336896 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.336867 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.338382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.338364 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.338593 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.338577 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.350228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.350201 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.350345 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.350264 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gv72\" (UniqueName: \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-kube-api-access-6gv72\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.438020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.437984 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.438243 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438064 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9354b4b-c613-48a1-aebf-f2eed6be09da-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.438243 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438209 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.438359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438266 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.438359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438342 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.438456 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438385 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.438456 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438424 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsksk\" (UniqueName: \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-kube-api-access-bsksk\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.438547 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438478 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.438644 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438625 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.438805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438730 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.438944 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438853 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.439009 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.438947 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9354b4b-c613-48a1-aebf-f2eed6be09da-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.439209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.439134 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.439209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.439154 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.441632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.441608 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.447606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.447579 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.456384 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.454063 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsksk\" (UniqueName: \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-kube-api-access-bsksk\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.456384 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.454210 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.494216 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.494160 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:35.542872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.542838 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:35.805532 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.805491 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g"] Oct 31 03:16:35.805669 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:16:35.805587 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711922a0_faeb_4427_87d6_c71c3e3ed2e9.slice/crio-03853c1ce3c664b051e69fb4b968627f455c5c97c2fa8484b3b162a6a0fa477a WatchSource:0}: Error finding container 03853c1ce3c664b051e69fb4b968627f455c5c97c2fa8484b3b162a6a0fa477a: Status 404 returned error can't find the container with id 03853c1ce3c664b051e69fb4b968627f455c5c97c2fa8484b3b162a6a0fa477a Oct 31 03:16:35.808514 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.808482 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:35.808609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.808545 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:35.808609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.808577 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:35.862898 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.861573 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5"] Oct 31 03:16:35.865533 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:16:35.865495 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9354b4b_c613_48a1_aebf_f2eed6be09da.slice/crio-05efd6266c1c1772b5f62ffc74605bc1b7434ddd38468ba3c175db7691362997 WatchSource:0}: Error finding container 05efd6266c1c1772b5f62ffc74605bc1b7434ddd38468ba3c175db7691362997: Status 404 returned error can't find the container with id 05efd6266c1c1772b5f62ffc74605bc1b7434ddd38468ba3c175db7691362997 Oct 31 03:16:35.868985 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.868943 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:35.869083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.869049 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:35.869128 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.869096 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:35.921180 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.921146 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" event={"ID":"711922a0-faeb-4427-87d6-c71c3e3ed2e9","Type":"ContainerStarted","Data":"7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93"} Oct 31 03:16:35.921321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.921204 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" event={"ID":"711922a0-faeb-4427-87d6-c71c3e3ed2e9","Type":"ContainerStarted","Data":"03853c1ce3c664b051e69fb4b968627f455c5c97c2fa8484b3b162a6a0fa477a"} Oct 31 03:16:35.923634 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.923606 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" event={"ID":"e9354b4b-c613-48a1-aebf-f2eed6be09da","Type":"ContainerStarted","Data":"05efd6266c1c1772b5f62ffc74605bc1b7434ddd38468ba3c175db7691362997"} Oct 31 03:16:35.956939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:35.956862 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" podStartSLOduration=0.956842424 podStartE2EDuration="956.842424ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:35.951286798 +0000 UTC m=+1277.631812926" watchObservedRunningTime="2026-10-31 03:16:35.956842424 +0000 UTC m=+1277.637368554" Oct 31 03:16:36.495677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:36.495641 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:36.497333 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:36.497301 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:16:36.497472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:36.497353 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:16:36.498158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:36.498134 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6980-7bae42"] Oct 31 03:16:36.498351 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:36.498326 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6980-7bae42" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" containerName="test-curl-6980-7bae42" containerID="cri-o://ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff" gracePeriod=2 Oct 31 03:16:36.501933 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:16:36.501897 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6980-7bae42\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:36.505468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:36.505440 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6980-7bae42"] Oct 31 03:16:36.975061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:36.961632 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" event={"ID":"e9354b4b-c613-48a1-aebf-f2eed6be09da","Type":"ContainerStarted","Data":"549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1"} Oct 31 03:16:37.001136 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:36.999950 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podStartSLOduration=1.9999172870000002 podStartE2EDuration="1.999917287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:36.999275335 +0000 UTC m=+1278.679801464" watchObservedRunningTime="2026-10-31 03:16:36.999917287 +0000 UTC m=+1278.680443408" Oct 31 03:16:37.004395 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:16:37.003305 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6980-7bae42\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:37.243450 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.243371 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs"] Oct 31 03:16:37.244205 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.244189 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" containerName="test-curl-6980-7bae42" Oct 31 03:16:37.244280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.244207 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" containerName="test-curl-6980-7bae42" Oct 31 03:16:37.244317 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.244299 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" containerName="test-curl-6980-7bae42" Oct 31 03:16:37.250739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.250714 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.270059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.262363 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:16:37.270059 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:16:37.262461 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6980-7bae42\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:37.270059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.262648 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:16:37.294062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.290607 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs"] Oct 31 03:16:37.401547 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.401492 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.401735 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.401620 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95l5x\" (UniqueName: \"kubernetes.io/projected/aa24c702-323b-41a0-a449-4682a705a5d6-kube-api-access-95l5x\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.401735 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.401662 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.496344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.496262 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:16:37.496344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.496318 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:16:37.504228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.504198 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.504359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.504293 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95l5x\" (UniqueName: \"kubernetes.io/projected/aa24c702-323b-41a0-a449-4682a705a5d6-kube-api-access-95l5x\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.504359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.504318 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.505026 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.504999 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.506706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.506687 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.520351 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.520326 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95l5x\" (UniqueName: \"kubernetes.io/projected/aa24c702-323b-41a0-a449-4682a705a5d6-kube-api-access-95l5x\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.543402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.543373 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:37.545460 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.545433 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:37.545561 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.545485 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:37.610465 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.610423 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:37.929465 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:16:37.929433 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa24c702_323b_41a0_a449_4682a705a5d6.slice/crio-4e72799c683f52fa54706f55a11c88285bd758805f89c6d5e434fb6f5664914f WatchSource:0}: Error finding container 4e72799c683f52fa54706f55a11c88285bd758805f89c6d5e434fb6f5664914f: Status 404 returned error can't find the container with id 4e72799c683f52fa54706f55a11c88285bd758805f89c6d5e434fb6f5664914f Oct 31 03:16:37.933020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.932988 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs"] Oct 31 03:16:37.992524 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:37.992487 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" event={"ID":"aa24c702-323b-41a0-a449-4682a705a5d6","Type":"ContainerStarted","Data":"4e72799c683f52fa54706f55a11c88285bd758805f89c6d5e434fb6f5664914f"} Oct 31 03:16:38.437608 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:16:38.437569 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6980-7bae42\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:38.496730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:38.496691 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:16:38.497170 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:38.496749 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:16:38.544760 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:38.544732 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:38.544864 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:38.544782 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:38.854270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:38.852240 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:38.928392 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:38.928359 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b521178c-351d-4c86-852c-8c92ef0fa0a0-kube-api-access-c97lc\" (UniqueName: \"kubernetes.io/projected/b521178c-351d-4c86-852c-8c92ef0fa0a0-kube-api-access-c97lc\") pod \"b521178c-351d-4c86-852c-8c92ef0fa0a0\" (UID: \"b521178c-351d-4c86-852c-8c92ef0fa0a0\") " Oct 31 03:16:38.930531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:38.930502 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b521178c-351d-4c86-852c-8c92ef0fa0a0-kube-api-access-c97lc" pod "b521178c-351d-4c86-852c-8c92ef0fa0a0" (UID: "b521178c-351d-4c86-852c-8c92ef0fa0a0"). InnerVolumeSpecName "kube-api-access-c97lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:16:39.005080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.003191 4456 generic.go:356] "Generic (PLEG): container finished" podID="b521178c-351d-4c86-852c-8c92ef0fa0a0" containerID="ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff" exitCode=137 Oct 31 03:16:39.005080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.003247 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:39.005080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.003289 4456 scope.go:122] "RemoveContainer" containerID="ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff" Oct 31 03:16:39.010085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.010049 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" event={"ID":"aa24c702-323b-41a0-a449-4682a705a5d6","Type":"ContainerStarted","Data":"fe56a063b78daa69a69c61dadfff36680fc7cb44845ffd2237bb72bf031950ab"} Oct 31 03:16:39.020565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.020545 4456 scope.go:122] "RemoveContainer" containerID="ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff" Oct 31 03:16:39.020854 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:16:39.020835 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff\": container with ID starting with ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff not found: ID does not exist" containerID="ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff" Oct 31 03:16:39.020910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.020863 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff"} err="failed to get container status \"ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff\": rpc error: code = NotFound desc = could not find container \"ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff\": container with ID starting with ca4ebd30fefeeae44d9240d52eb75cd4eb57cb36ba21ff774452ce357cb585ff not found: ID does not exist" Oct 31 03:16:39.031001 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.030976 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c97lc\" (UniqueName: \"kubernetes.io/projected/b521178c-351d-4c86-852c-8c92ef0fa0a0-kube-api-access-c97lc\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.036550 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:16:39.036525 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6980-7bae42\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:39.037599 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.037536 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" podStartSLOduration=2.0375206280000002 podStartE2EDuration="2.037520628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:39.033076818 +0000 UTC m=+1280.713602947" watchObservedRunningTime="2026-10-31 03:16:39.037520628 +0000 UTC m=+1280.718046756" Oct 31 03:16:39.496732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.496631 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:16:39.496732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.496704 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:16:39.544260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.544227 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:39.544437 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:39.544279 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:40.039133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.039099 4456 generic.go:356] "Generic (PLEG): container finished" podID="aa24c702-323b-41a0-a449-4682a705a5d6" containerID="fe56a063b78daa69a69c61dadfff36680fc7cb44845ffd2237bb72bf031950ab" exitCode=0 Oct 31 03:16:40.039133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.039140 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" event={"ID":"aa24c702-323b-41a0-a449-4682a705a5d6","Type":"ContainerDied","Data":"fe56a063b78daa69a69c61dadfff36680fc7cb44845ffd2237bb72bf031950ab"} Oct 31 03:16:40.042848 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:16:40.042814 4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6980-7bae42\" is forbidden: User \"system:node:ip-10-0-129-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-107.us-east-2.compute.internal' and this object" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" pod="opendatahub/test-curl-6980-7bae42" Oct 31 03:16:40.438373 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.438336 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b521178c-351d-4c86-852c-8c92ef0fa0a0" path="/var/lib/kubelet/pods/b521178c-351d-4c86-852c-8c92ef0fa0a0/volumes" Oct 31 03:16:40.496382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.496343 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:16:40.496576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.496413 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:16:40.521558 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.521525 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6"] Oct 31 03:16:40.526897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.526877 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.541836 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.540794 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6"] Oct 31 03:16:40.544172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.544142 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:40.544307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.544199 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:40.576056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.558579 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b"] Oct 31 03:16:40.611891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.611862 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g"] Oct 31 03:16:40.612292 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.611988 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.618730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.618696 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b"] Oct 31 03:16:40.618841 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.618812 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.621635 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.621612 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g"] Oct 31 03:16:40.711006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.710910 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25526fb-70e1-44f2-8157-c4648fc91e9a-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.711006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.710968 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7526b30d-815e-4669-808d-2bf77dbfbc8a-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.711006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.710987 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.711006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711006 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711028 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711109 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711137 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711167 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0478a844-cbe2-41b0-a4fe-78fcef28831d-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711194 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711215 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711241 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711283 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711314 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711330 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.711386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711374 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711395 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711463 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frwt\" (UniqueName: \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-kube-api-access-8frwt\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711489 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711517 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ftvf\" (UniqueName: \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-kube-api-access-5ftvf\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711540 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711556 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711581 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711604 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711620 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711670 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711701 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.711995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.711717 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzhs\" (UniqueName: \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-kube-api-access-8lzhs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.812968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.812932 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.812968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.812970 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzhs\" (UniqueName: \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-kube-api-access-8lzhs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.813253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813014 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25526fb-70e1-44f2-8157-c4648fc91e9a-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.813253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813082 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7526b30d-815e-4669-808d-2bf77dbfbc8a-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.813253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813107 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.813253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813137 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.813253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813169 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.813253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813260 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.813646 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813310 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.813741 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813717 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.813741 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813733 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.813860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813793 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.813860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813804 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25526fb-70e1-44f2-8157-c4648fc91e9a-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.813860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813833 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0478a844-cbe2-41b0-a4fe-78fcef28831d-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.814030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813873 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.814030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813906 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.814030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.813947 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.814030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814010 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.814305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814101 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.814305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814128 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.814305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814193 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.814305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814226 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.814305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814261 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.814580 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814337 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frwt\" (UniqueName: \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-kube-api-access-8frwt\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.814580 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814378 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.814580 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814532 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.814751 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814589 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0478a844-cbe2-41b0-a4fe-78fcef28831d-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.814812 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814781 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.814872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814827 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.814872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814850 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.814962 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814909 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftvf\" (UniqueName: \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-kube-api-access-5ftvf\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.815023 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814963 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.815224 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.814988 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.815345 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.815299 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.815607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.815241 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.815690 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.815654 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.815690 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.815681 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.815807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.815758 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.815865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.815830 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7526b30d-815e-4669-808d-2bf77dbfbc8a-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.815991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.815970 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.816177 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.816154 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.816275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.816253 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.818340 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.818301 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.819308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.818443 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.819308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.818494 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.819308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.818545 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.819308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.818739 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.819683 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.819665 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.826685 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.826662 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.826883 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.826857 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzhs\" (UniqueName: \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-kube-api-access-8lzhs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.827446 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.827426 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.827579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.827502 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:40.827988 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.827967 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frwt\" (UniqueName: \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-kube-api-access-8frwt\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:40.829133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.829108 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ftvf\" (UniqueName: \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-kube-api-access-5ftvf\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.878063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.872820 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:40.989109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:40.960351 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:41.023591 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.023401 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:41.243433 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.240211 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6"] Oct 31 03:16:41.243433 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:16:41.240731 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7526b30d_815e_4669_808d_2bf77dbfbc8a.slice/crio-af663ed61b52af130b9713ab11f1fb7b6d818ae4767d150bc902f41680e392b1 WatchSource:0}: Error finding container af663ed61b52af130b9713ab11f1fb7b6d818ae4767d150bc902f41680e392b1: Status 404 returned error can't find the container with id af663ed61b52af130b9713ab11f1fb7b6d818ae4767d150bc902f41680e392b1 Oct 31 03:16:41.244755 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.244710 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:41.244884 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.244803 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:41.244884 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.244848 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:41.498259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.496828 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:16:41.498259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.496877 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:16:41.499976 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.499416 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b"] Oct 31 03:16:41.508816 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:16:41.508719 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25526fb_70e1_44f2_8157_c4648fc91e9a.slice/crio-f5975003b7c614237b2dbecf03bf9026167b6eb7f34d1e14aad5a522ffb8552d WatchSource:0}: Error finding container f5975003b7c614237b2dbecf03bf9026167b6eb7f34d1e14aad5a522ffb8552d: Status 404 returned error can't find the container with id f5975003b7c614237b2dbecf03bf9026167b6eb7f34d1e14aad5a522ffb8552d Oct 31 03:16:41.516865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.516830 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:41.516982 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.516914 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:41.516982 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.516960 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:41.527112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.526215 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:41.531607 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:16:41.530510 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0478a844_cbe2_41b0_a4fe_78fcef28831d.slice/crio-97af4a15c6833b893fbadb64c3995d64e0b77eb7033479ca80ce0500b6f737bf WatchSource:0}: Error finding container 97af4a15c6833b893fbadb64c3995d64e0b77eb7033479ca80ce0500b6f737bf: Status 404 returned error can't find the container with id 97af4a15c6833b893fbadb64c3995d64e0b77eb7033479ca80ce0500b6f737bf Oct 31 03:16:41.534947 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.534920 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g"] Oct 31 03:16:41.544353 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.544316 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:41.544493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.544399 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:41.544493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.544450 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:16:41.604023 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.593431 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:41.604023 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.593758 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:41.641198 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.641129 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-openshift-ca" pod "aa24c702-323b-41a0-a449-4682a705a5d6" (UID: "aa24c702-323b-41a0-a449-4682a705a5d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:41.662247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.660330 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-openshift-ca\") pod \"aa24c702-323b-41a0-a449-4682a705a5d6\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " Oct 31 03:16:41.662247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.660493 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-xks-ca\") pod \"aa24c702-323b-41a0-a449-4682a705a5d6\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " Oct 31 03:16:41.662247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.660533 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa24c702-323b-41a0-a449-4682a705a5d6-kube-api-access-95l5x\" (UniqueName: \"kubernetes.io/projected/aa24c702-323b-41a0-a449-4682a705a5d6-kube-api-access-95l5x\") pod \"aa24c702-323b-41a0-a449-4682a705a5d6\" (UID: \"aa24c702-323b-41a0-a449-4682a705a5d6\") " Oct 31 03:16:41.734419 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.734328 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-xks-ca" pod "aa24c702-323b-41a0-a449-4682a705a5d6" (UID: "aa24c702-323b-41a0-a449-4682a705a5d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:41.735058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.734993 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa24c702-323b-41a0-a449-4682a705a5d6-kube-api-access-95l5x" pod "aa24c702-323b-41a0-a449-4682a705a5d6" (UID: "aa24c702-323b-41a0-a449-4682a705a5d6"). InnerVolumeSpecName "kube-api-access-95l5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:16:41.793978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.793863 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:41.793978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.793919 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa24c702-323b-41a0-a449-4682a705a5d6-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:41.793978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:41.793936 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95l5x\" (UniqueName: \"kubernetes.io/projected/aa24c702-323b-41a0-a449-4682a705a5d6-kube-api-access-95l5x\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:42.145851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.140619 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" event={"ID":"7526b30d-815e-4669-808d-2bf77dbfbc8a","Type":"ContainerStarted","Data":"610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4"} Oct 31 03:16:42.145851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.140686 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" event={"ID":"7526b30d-815e-4669-808d-2bf77dbfbc8a","Type":"ContainerStarted","Data":"af663ed61b52af130b9713ab11f1fb7b6d818ae4767d150bc902f41680e392b1"} Oct 31 03:16:42.156996 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.156960 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" event={"ID":"f25526fb-70e1-44f2-8157-c4648fc91e9a","Type":"ContainerStarted","Data":"8fc96c5e205efbe0f587544fb0508534febab2d7a456edbcbe539b5e1debcdbd"} Oct 31 03:16:42.157377 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.157345 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" event={"ID":"f25526fb-70e1-44f2-8157-c4648fc91e9a","Type":"ContainerStarted","Data":"f5975003b7c614237b2dbecf03bf9026167b6eb7f34d1e14aad5a522ffb8552d"} Oct 31 03:16:42.163005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.162966 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" event={"ID":"0478a844-cbe2-41b0-a4fe-78fcef28831d","Type":"ContainerStarted","Data":"67b00222b91788090798f50a5561e588df911e1bb25e47bdd45d8dcec276d5c5"} Oct 31 03:16:42.164805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.164779 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" event={"ID":"0478a844-cbe2-41b0-a4fe-78fcef28831d","Type":"ContainerStarted","Data":"97af4a15c6833b893fbadb64c3995d64e0b77eb7033479ca80ce0500b6f737bf"} Oct 31 03:16:42.166308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.166279 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" event={"ID":"aa24c702-323b-41a0-a449-4682a705a5d6","Type":"ContainerDied","Data":"4e72799c683f52fa54706f55a11c88285bd758805f89c6d5e434fb6f5664914f"} Oct 31 03:16:42.166479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.166463 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e72799c683f52fa54706f55a11c88285bd758805f89c6d5e434fb6f5664914f" Oct 31 03:16:42.166643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.166628 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs" Oct 31 03:16:42.183208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.183140 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" podStartSLOduration=2.183110738 podStartE2EDuration="2.183110738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:42.177241213 +0000 UTC m=+1283.857767340" watchObservedRunningTime="2026-10-31 03:16:42.183110738 +0000 UTC m=+1283.863636867" Oct 31 03:16:42.253506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.253441 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podStartSLOduration=2.253421021 podStartE2EDuration="2.253421021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:42.210589663 +0000 UTC m=+1283.891115791" watchObservedRunningTime="2026-10-31 03:16:42.253421021 +0000 UTC m=+1283.933947146" Oct 31 03:16:42.335070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.326592 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podStartSLOduration=2.326573493 podStartE2EDuration="2.326573493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:42.2475027 +0000 UTC m=+1283.928028863" watchObservedRunningTime="2026-10-31 03:16:42.326573493 +0000 UTC m=+1284.007099617" Oct 31 03:16:42.502003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.501967 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:16:42.523403 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.523359 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:16:42.553060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.543612 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:42.553904 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.553863 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:42.915931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.915890 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:42.916342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.916314 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:16:42.916511 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.916488 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:16:42.964156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.962199 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:42.964457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.964099 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:16:42.964630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:42.964597 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:16:43.033688 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.033653 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:43.035640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.035609 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:16:43.035856 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.035832 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:16:43.897877 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.881224 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:43.897877 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.881291 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:43.897877 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.881688 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:16:43.928687 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.926541 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:43.928687 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.927934 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:43.938534 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.930059 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:16:43.940726 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.939518 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:16:43.940726 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.940463 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:16:43.947081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.941222 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 03:16:43.947081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:43.941519 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 31 03:16:44.014072 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.013867 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:44.076070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.075137 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:16:44.076070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.075201 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:16:44.124214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.110127 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:16:44.124214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.110187 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:16:44.142227 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.124452 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:16:44.142227 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.124514 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:16:44.175138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.142715 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:44.375063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.360264 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:44.469300 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.464466 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:44.651348 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.614234 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:44.651680 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.651642 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:44.890117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.889296 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:44.890287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.890135 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:16:44.902065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.891550 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:16:44.902065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.891591 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:16:44.969061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.944253 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:44.969061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.946293 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 03:16:44.969061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.946678 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": context deadline exceeded" Oct 31 03:16:44.969061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.947202 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:16:44.969061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.947296 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:16:45.044716 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:44.987522 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:16:45.133596 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.133549 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:16:45.133813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.014809 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:45.134280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.014725 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:45.134720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.095579 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:16:45.134911 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.134874 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:16:45.135381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.028471 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:45.135798 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.135777 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:16:45.135968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.135955 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:16:45.136108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.136096 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:16:45.149196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.149109 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:16:45.157073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.156660 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:16:45.485434 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.472508 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 03:16:45.496061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.494184 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:16:45.684387 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.682427 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:16:45.758535 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.758447 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:45.758765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.758736 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:45.773308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.773277 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:16:45.928983 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.928933 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:45.960514 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:45.960472 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 31 03:16:46.003126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.003091 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:16:46.003450 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.003422 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:16:46.003967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.003360 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:16:46.004135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.004112 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:16:46.068321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.068210 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:16:46.112069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.093524 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:16:46.255164 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.137545 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:46.261381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.137578 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:16:46.302591 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.164014 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:46.511513 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.511464 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 03:16:46.556060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.553799 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:46.556060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.553863 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:46.692367 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.692196 4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:46.885591 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.885498 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:16:46.885841 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.885810 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:16:46.960971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.960923 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.016617 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.980782 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:16:47.020140 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.020091 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:16:47.037351 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:46.995249 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.038160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.031142 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:16:47.038363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.038338 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:16:47.265368 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.265323 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.302840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.302794 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.306831 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.303988 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:16:47.342063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.337476 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:16:47.575115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.556460 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.575115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.558462 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:47.575115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.558508 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:47.692193 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.692145 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.812066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.801645 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 03:16:47.907874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.907826 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:16:47.935171 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.935125 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:16:48.102070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:47.992170 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:16:48.102070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.089553 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:16:48.102070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.024254 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.102070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.091134 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.173414 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.173098 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:16:48.173826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.173643 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:16:48.269365 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.269320 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.438905 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.438194 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.562110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.560711 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.711119 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.710736 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.759061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.753769 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:48.759061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.753848 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:48.886828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.886793 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:16:48.887135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.887094 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:16:48.977354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.977258 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb"] Oct 31 03:16:48.990061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.990007 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd"] Oct 31 03:16:48.990334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.990315 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:16:48.990506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.990494 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:16:48.991571 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.983507 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:16:48.991835 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:48.991807 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:16:49.025863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.025827 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:16:49.026115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.026085 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:16:49.114206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.114157 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:16:49.140918 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.140153 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:49.521385 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.521333 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:49.524435 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.524398 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:16:49.568094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.552776 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:49.568094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.552838 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:49.807188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.807083 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 03:16:49.809632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.809597 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 03:16:49.810452 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.810277 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 03:16:49.880628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.880447 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:16:49.882813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.882788 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:16:49.883457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.883430 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:16:49.883654 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.883621 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:16:49.995079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.985685 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:16:49.995079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:49.985753 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:16:50.100456 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:50.095967 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:16:50.100456 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:50.096060 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:16:50.130211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:50.128623 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:16:50.300265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:50.292728 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:50.346530 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:50.346474 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:50.553814 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:50.553771 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:16:50.564685 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:50.564652 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:16:50.564930 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:50.564905 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:16:50.565469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:50.554065 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:16:50.818152 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:50.818022 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 03:16:51.054158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.054125 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:16:51.054440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.054412 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:16:51.085074 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.080953 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:16:51.085346 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.085313 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:16:51.274254 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.273925 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:51.282725 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.282692 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:16:51.326415 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.326382 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:51.326701 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.326690 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:16:51.326840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.326831 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:16:51.328359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.301024 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:51.331299 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.331232 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:51.337901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.337834 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:16:51.411204 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.411161 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:51.411969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.411943 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:16:51.460065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.452219 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:16:51.510350 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.510308 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:16:51.575748 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.575712 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:16:51.601871 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.601765 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:51.604546 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.604508 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:16:51.715833 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.715801 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:16:51.736290 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:51.736242 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:16:52.025689 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:52.025606 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:16:52.025954 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:52.025924 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:16:52.042303 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:52.042256 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:52.043076 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:52.043051 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:52.044598 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:52.044544 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:16:52.607972 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:52.607929 4456 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:16:52.608934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:52.608604 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:52.682557 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:52.682526 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:52.697978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:52.697947 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:16:53.030257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:53.030220 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:53.031982 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:53.031958 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:53.032805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:53.032782 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:16:53.993392 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:53.993340 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" podUID="6943cde0-73dd-4c09-b335-531048357951" containerName="payload-pre-processing" containerID="cri-o://e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a" gracePeriod=25 Oct 31 03:16:53.993889 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:53.993361 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" podUID="18c29c9c-1fd0-47c9-bd79-0bbffb9e3902" containerName="payload-processing" containerID="cri-o://3634a9b0dd499b352d81f727f10277d392f7e978f3ba33026ff86b2d8307d729" gracePeriod=25 Oct 31 03:16:59.439630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:16:59.439595 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" podUID="6943cde0-73dd-4c09-b335-531048357951" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:17:00.456659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.456614 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" podUID="18c29c9c-1fd0-47c9-bd79-0bbffb9e3902" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:17:00.469032 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.468997 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx"] Oct 31 03:17:00.470292 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.470266 4456 cpu_manager.go:414] "containerMap: removing container" podUID="aa24c702-323b-41a0-a449-4682a705a5d6" containerName="revoke-keys" Oct 31 03:17:00.470389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.470294 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa24c702-323b-41a0-a449-4682a705a5d6" containerName="revoke-keys" Oct 31 03:17:00.470578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.470565 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa24c702-323b-41a0-a449-4682a705a5d6" containerName="revoke-keys" Oct 31 03:17:00.477614 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.477028 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:00.482314 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.482288 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:17:00.482945 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.482915 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx"] Oct 31 03:17:00.484475 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.484147 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:17:00.550668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.550630 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:00.550914 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.550881 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snszr\" (UniqueName: \"kubernetes.io/projected/5ad03736-abe0-4140-bcea-5b915b72269b-kube-api-access-snszr\") pod \"maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:00.551325 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.551306 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:00.653503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.653469 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:00.653503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.653515 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snszr\" (UniqueName: \"kubernetes.io/projected/5ad03736-abe0-4140-bcea-5b915b72269b-kube-api-access-snszr\") pod \"maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:00.653760 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.653685 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:00.654501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.654473 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:00.662287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.662253 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:00.668593 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.668562 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snszr\" (UniqueName: \"kubernetes.io/projected/5ad03736-abe0-4140-bcea-5b915b72269b-kube-api-access-snszr\") pod \"maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:00.798085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:00.797967 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:01.109914 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:17:01.109865 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad03736_abe0_4140_bcea_5b915b72269b.slice/crio-4e001149d3c35fa0348ae81c50cd7788e7801b9fcc84d54dcd5a281be8633832 WatchSource:0}: Error finding container 4e001149d3c35fa0348ae81c50cd7788e7801b9fcc84d54dcd5a281be8633832: Status 404 returned error can't find the container with id 4e001149d3c35fa0348ae81c50cd7788e7801b9fcc84d54dcd5a281be8633832 Oct 31 03:17:01.112423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:01.112394 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx"] Oct 31 03:17:01.712285 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:01.712247 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" event={"ID":"5ad03736-abe0-4140-bcea-5b915b72269b","Type":"ContainerStarted","Data":"01e57bf5f286b1a37071eba54480fa800c717ac33da138a852e0ab0cb6ccb0f5"} Oct 31 03:17:01.712806 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:01.712300 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" event={"ID":"5ad03736-abe0-4140-bcea-5b915b72269b","Type":"ContainerStarted","Data":"4e001149d3c35fa0348ae81c50cd7788e7801b9fcc84d54dcd5a281be8633832"} Oct 31 03:17:01.733764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:01.733711 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" podStartSLOduration=1.73369718 podStartE2EDuration="1.73369718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:01.732187253 +0000 UTC m=+1303.412713381" watchObservedRunningTime="2026-10-31 03:17:01.73369718 +0000 UTC m=+1303.414223351" Oct 31 03:17:02.135973 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:17:02.135928 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:17:02.731792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:02.731754 4456 generic.go:356] "Generic (PLEG): container finished" podID="5ad03736-abe0-4140-bcea-5b915b72269b" containerID="01e57bf5f286b1a37071eba54480fa800c717ac33da138a852e0ab0cb6ccb0f5" exitCode=0 Oct 31 03:17:02.732307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:02.731844 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" event={"ID":"5ad03736-abe0-4140-bcea-5b915b72269b","Type":"ContainerDied","Data":"01e57bf5f286b1a37071eba54480fa800c717ac33da138a852e0ab0cb6ccb0f5"} Oct 31 03:17:03.248375 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.248337 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s"] Oct 31 03:17:03.253566 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.253540 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.264730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.264694 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s"] Oct 31 03:17:03.315495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.315461 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xct6h\" (UniqueName: \"kubernetes.io/projected/5c62f1ec-97fb-4f46-991c-9088c99e11aa-kube-api-access-xct6h\") pod \"maas-api-revoke-sub-863735a9-9mx5s\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.315791 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.315752 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863735a9-9mx5s\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.315899 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.315882 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863735a9-9mx5s\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.417445 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.417405 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863735a9-9mx5s\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.417643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.417490 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863735a9-9mx5s\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.417643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.417625 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct6h\" (UniqueName: \"kubernetes.io/projected/5c62f1ec-97fb-4f46-991c-9088c99e11aa-kube-api-access-xct6h\") pod \"maas-api-revoke-sub-863735a9-9mx5s\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.418297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.418273 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863735a9-9mx5s\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.420005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.419979 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863735a9-9mx5s\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.430656 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.430631 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xct6h\" (UniqueName: \"kubernetes.io/projected/5c62f1ec-97fb-4f46-991c-9088c99e11aa-kube-api-access-xct6h\") pod \"maas-api-revoke-sub-863735a9-9mx5s\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.576608 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.576525 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:03.907262 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:17:03.907225 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c62f1ec_97fb_4f46_991c_9088c99e11aa.slice/crio-c21ce2f6712c2231591fb2f76fb8b226cb44dfb8df243124d9eb10e39f62850a WatchSource:0}: Error finding container c21ce2f6712c2231591fb2f76fb8b226cb44dfb8df243124d9eb10e39f62850a: Status 404 returned error can't find the container with id c21ce2f6712c2231591fb2f76fb8b226cb44dfb8df243124d9eb10e39f62850a Oct 31 03:17:03.909662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.909640 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s"] Oct 31 03:17:03.932024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:03.932005 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:04.033051 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.033008 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-openshift-ca\") pod \"5ad03736-abe0-4140-bcea-5b915b72269b\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " Oct 31 03:17:04.033236 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.033144 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-xks-ca\") pod \"5ad03736-abe0-4140-bcea-5b915b72269b\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " Oct 31 03:17:04.033371 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.033342 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad03736-abe0-4140-bcea-5b915b72269b-kube-api-access-snszr\" (UniqueName: \"kubernetes.io/projected/5ad03736-abe0-4140-bcea-5b915b72269b-kube-api-access-snszr\") pod \"5ad03736-abe0-4140-bcea-5b915b72269b\" (UID: \"5ad03736-abe0-4140-bcea-5b915b72269b\") " Oct 31 03:17:04.033426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.033394 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-openshift-ca" pod "5ad03736-abe0-4140-bcea-5b915b72269b" (UID: "5ad03736-abe0-4140-bcea-5b915b72269b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:04.034161 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.034135 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:04.046835 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.046806 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-xks-ca" pod "5ad03736-abe0-4140-bcea-5b915b72269b" (UID: "5ad03736-abe0-4140-bcea-5b915b72269b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:04.046953 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.046927 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad03736-abe0-4140-bcea-5b915b72269b-kube-api-access-snszr" pod "5ad03736-abe0-4140-bcea-5b915b72269b" (UID: "5ad03736-abe0-4140-bcea-5b915b72269b"). InnerVolumeSpecName "kube-api-access-snszr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:04.136272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.136240 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad03736-abe0-4140-bcea-5b915b72269b-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:04.136272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.136268 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snszr\" (UniqueName: \"kubernetes.io/projected/5ad03736-abe0-4140-bcea-5b915b72269b-kube-api-access-snszr\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:04.755802 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.755720 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" Oct 31 03:17:04.755991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.755729 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx" event={"ID":"5ad03736-abe0-4140-bcea-5b915b72269b","Type":"ContainerDied","Data":"4e001149d3c35fa0348ae81c50cd7788e7801b9fcc84d54dcd5a281be8633832"} Oct 31 03:17:04.756159 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.756021 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e001149d3c35fa0348ae81c50cd7788e7801b9fcc84d54dcd5a281be8633832" Oct 31 03:17:04.758545 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.758512 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" event={"ID":"5c62f1ec-97fb-4f46-991c-9088c99e11aa","Type":"ContainerStarted","Data":"26baca26b51a5cdb8261be6c6ec18297f155b82012e3e223d7ff7c880ef7e746"} Oct 31 03:17:04.758681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.758570 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" event={"ID":"5c62f1ec-97fb-4f46-991c-9088c99e11aa","Type":"ContainerStarted","Data":"c21ce2f6712c2231591fb2f76fb8b226cb44dfb8df243124d9eb10e39f62850a"} Oct 31 03:17:04.784484 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:04.784420 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" podStartSLOduration=1.784407106 podStartE2EDuration="1.784407106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:04.783110661 +0000 UTC m=+1306.463636789" watchObservedRunningTime="2026-10-31 03:17:04.784407106 +0000 UTC m=+1306.464933233" Oct 31 03:17:05.791495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:05.791450 4456 generic.go:356] "Generic (PLEG): container finished" podID="5c62f1ec-97fb-4f46-991c-9088c99e11aa" containerID="26baca26b51a5cdb8261be6c6ec18297f155b82012e3e223d7ff7c880ef7e746" exitCode=0 Oct 31 03:17:05.792104 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:05.792073 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" event={"ID":"5c62f1ec-97fb-4f46-991c-9088c99e11aa","Type":"ContainerDied","Data":"26baca26b51a5cdb8261be6c6ec18297f155b82012e3e223d7ff7c880ef7e746"} Oct 31 03:17:07.116716 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.116691 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:07.231466 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.231429 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-xks-ca\") pod \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " Oct 31 03:17:07.231658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.231642 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c62f1ec-97fb-4f46-991c-9088c99e11aa-kube-api-access-xct6h\" (UniqueName: \"kubernetes.io/projected/5c62f1ec-97fb-4f46-991c-9088c99e11aa-kube-api-access-xct6h\") pod \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " Oct 31 03:17:07.231835 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.231813 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-openshift-ca\") pod \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\" (UID: \"5c62f1ec-97fb-4f46-991c-9088c99e11aa\") " Oct 31 03:17:07.232220 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.232189 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-openshift-ca" pod "5c62f1ec-97fb-4f46-991c-9088c99e11aa" (UID: "5c62f1ec-97fb-4f46-991c-9088c99e11aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:07.232988 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.232958 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:07.234615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.234573 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-xks-ca" pod "5c62f1ec-97fb-4f46-991c-9088c99e11aa" (UID: "5c62f1ec-97fb-4f46-991c-9088c99e11aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:07.235604 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.235578 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c62f1ec-97fb-4f46-991c-9088c99e11aa-kube-api-access-xct6h" pod "5c62f1ec-97fb-4f46-991c-9088c99e11aa" (UID: "5c62f1ec-97fb-4f46-991c-9088c99e11aa"). InnerVolumeSpecName "kube-api-access-xct6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:07.354117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.336258 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c62f1ec-97fb-4f46-991c-9088c99e11aa-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:07.354117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.336303 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xct6h\" (UniqueName: \"kubernetes.io/projected/5c62f1ec-97fb-4f46-991c-9088c99e11aa-kube-api-access-xct6h\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:07.909059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.906789 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" event={"ID":"5c62f1ec-97fb-4f46-991c-9088c99e11aa","Type":"ContainerDied","Data":"c21ce2f6712c2231591fb2f76fb8b226cb44dfb8df243124d9eb10e39f62850a"} Oct 31 03:17:07.909059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.906864 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c21ce2f6712c2231591fb2f76fb8b226cb44dfb8df243124d9eb10e39f62850a" Oct 31 03:17:07.909059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:07.906951 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s" Oct 31 03:17:09.450062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:09.447135 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" podUID="6943cde0-73dd-4c09-b335-531048357951" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:17:10.532077 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:10.522211 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" podUID="18c29c9c-1fd0-47c9-bd79-0bbffb9e3902" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:17:11.103135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:11.087432 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs"] Oct 31 03:17:11.512958 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:11.512918 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:17:11.572059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:11.567782 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gdw" podUID="4f1f2ef9-d74d-4c50-8eeb-60103414adaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:14.340837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.340809 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:17:14.405745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.405711 4456 generic.go:356] "Generic (PLEG): container finished" podID="18c29c9c-1fd0-47c9-bd79-0bbffb9e3902" containerID="3634a9b0dd499b352d81f727f10277d392f7e978f3ba33026ff86b2d8307d729" exitCode=0 Oct 31 03:17:14.405918 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.405825 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" event={"ID":"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902","Type":"ContainerDied","Data":"3634a9b0dd499b352d81f727f10277d392f7e978f3ba33026ff86b2d8307d729"} Oct 31 03:17:14.408478 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.408449 4456 generic.go:356] "Generic (PLEG): container finished" podID="6943cde0-73dd-4c09-b335-531048357951" containerID="e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a" exitCode=0 Oct 31 03:17:14.408592 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.408521 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" event={"ID":"6943cde0-73dd-4c09-b335-531048357951","Type":"ContainerDied","Data":"e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a"} Oct 31 03:17:14.408592 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.408575 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" event={"ID":"6943cde0-73dd-4c09-b335-531048357951","Type":"ContainerDied","Data":"48f58e0fe0bb7cb3e3c704ad64ecfa0a625aaeec5abb23a6bfc2001669248788"} Oct 31 03:17:14.408672 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.408604 4456 scope.go:122] "RemoveContainer" containerID="e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a" Oct 31 03:17:14.408775 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.408759 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd" Oct 31 03:17:14.428986 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.428963 4456 scope.go:122] "RemoveContainer" containerID="e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a" Oct 31 03:17:14.429358 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:17:14.429318 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a\": container with ID starting with e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a not found: ID does not exist" containerID="e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a" Oct 31 03:17:14.429481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.429355 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a"} err="failed to get container status \"e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a\": rpc error: code = NotFound desc = could not find container \"e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a\": container with ID starting with e370ccb19e95fb9679e2d6424e05949d47d1ee2671f2596c79ccae8a5229266a not found: ID does not exist" Oct 31 03:17:14.433122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.433103 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:17:14.435714 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.435695 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6943cde0-73dd-4c09-b335-531048357951-kube-api-access-hlbjv\" (UniqueName: \"kubernetes.io/projected/6943cde0-73dd-4c09-b335-531048357951-kube-api-access-hlbjv\") pod \"6943cde0-73dd-4c09-b335-531048357951\" (UID: \"6943cde0-73dd-4c09-b335-531048357951\") " Oct 31 03:17:14.435812 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.435800 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6943cde0-73dd-4c09-b335-531048357951-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6943cde0-73dd-4c09-b335-531048357951-plugins-config-volume\") pod \"6943cde0-73dd-4c09-b335-531048357951\" (UID: \"6943cde0-73dd-4c09-b335-531048357951\") " Oct 31 03:17:14.436274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.436246 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6943cde0-73dd-4c09-b335-531048357951-plugins-config-volume" pod "6943cde0-73dd-4c09-b335-531048357951" (UID: "6943cde0-73dd-4c09-b335-531048357951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:14.436608 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.436588 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6943cde0-73dd-4c09-b335-531048357951-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:14.438118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.438096 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6943cde0-73dd-4c09-b335-531048357951-kube-api-access-hlbjv" pod "6943cde0-73dd-4c09-b335-531048357951" (UID: "6943cde0-73dd-4c09-b335-531048357951"). InnerVolumeSpecName "kube-api-access-hlbjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:14.441389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.440442 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x"] Oct 31 03:17:14.441389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.440633 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" containerID="cri-o://4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a" gracePeriod=30 Oct 31 03:17:14.537952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.537921 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-plugins-config-volume\") pod \"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902\" (UID: \"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902\") " Oct 31 03:17:14.538342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.538322 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-kube-api-access-lfrbr\" (UniqueName: \"kubernetes.io/projected/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-kube-api-access-lfrbr\") pod \"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902\" (UID: \"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902\") " Oct 31 03:17:14.538456 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.538396 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-plugins-config-volume" pod "18c29c9c-1fd0-47c9-bd79-0bbffb9e3902" (UID: "18c29c9c-1fd0-47c9-bd79-0bbffb9e3902"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:14.539233 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.539203 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:14.539233 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.539232 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlbjv\" (UniqueName: \"kubernetes.io/projected/6943cde0-73dd-4c09-b335-531048357951-kube-api-access-hlbjv\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:14.540550 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.540526 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-kube-api-access-lfrbr" pod "18c29c9c-1fd0-47c9-bd79-0bbffb9e3902" (UID: "18c29c9c-1fd0-47c9-bd79-0bbffb9e3902"). InnerVolumeSpecName "kube-api-access-lfrbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:14.641272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.641192 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfrbr\" (UniqueName: \"kubernetes.io/projected/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902-kube-api-access-lfrbr\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:14.738225 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.738191 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd"] Oct 31 03:17:14.743605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:14.743578 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-jzksd"] Oct 31 03:17:15.419585 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:15.419559 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" Oct 31 03:17:15.420118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:15.419591 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb" event={"ID":"18c29c9c-1fd0-47c9-bd79-0bbffb9e3902","Type":"ContainerDied","Data":"511bff099b7c7ea14950b133062bb343e756566a2dfeba66149efbae2a523458"} Oct 31 03:17:15.420118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:15.419655 4456 scope.go:122] "RemoveContainer" containerID="3634a9b0dd499b352d81f727f10277d392f7e978f3ba33026ff86b2d8307d729" Oct 31 03:17:15.455699 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:15.455640 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb"] Oct 31 03:17:15.463258 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:15.463238 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-xjxhb"] Oct 31 03:17:16.088735 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.088692 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" podUID="7fca5e6e-5a1b-45f8-838e-1fa234468a78" containerName="payload-pre-processing" containerID="cri-o://ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8" gracePeriod=25 Oct 31 03:17:16.431928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.431888 4456 generic.go:356] "Generic (PLEG): container finished" podID="7fca5e6e-5a1b-45f8-838e-1fa234468a78" containerID="ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8" exitCode=0 Oct 31 03:17:16.433525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.433500 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:17:16.438934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.438912 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c29c9c-1fd0-47c9-bd79-0bbffb9e3902" path="/var/lib/kubelet/pods/18c29c9c-1fd0-47c9-bd79-0bbffb9e3902/volumes" Oct 31 03:17:16.439587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.439568 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6943cde0-73dd-4c09-b335-531048357951" path="/var/lib/kubelet/pods/6943cde0-73dd-4c09-b335-531048357951/volumes" Oct 31 03:17:16.440028 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.440006 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" event={"ID":"7fca5e6e-5a1b-45f8-838e-1fa234468a78","Type":"ContainerDied","Data":"ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8"} Oct 31 03:17:16.440109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.440084 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" event={"ID":"7fca5e6e-5a1b-45f8-838e-1fa234468a78","Type":"ContainerDied","Data":"5b8a540db4fc5b29eb16d980c5df78750ab2d5aa56e153a8a2d8bcf95ab0c723"} Oct 31 03:17:16.440153 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.440117 4456 scope.go:122] "RemoveContainer" containerID="ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8" Oct 31 03:17:16.453691 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.453670 4456 scope.go:122] "RemoveContainer" containerID="ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8" Oct 31 03:17:16.453973 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:17:16.453947 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8\": container with ID starting with ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8 not found: ID does not exist" containerID="ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8" Oct 31 03:17:16.454081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.453985 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8"} err="failed to get container status \"ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8\": rpc error: code = NotFound desc = could not find container \"ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8\": container with ID starting with ee9816da43406c8ef96afd7186579477aa0d775bb703150cffcd65917181cdb8 not found: ID does not exist" Oct 31 03:17:16.487662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.487622 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fca5e6e-5a1b-45f8-838e-1fa234468a78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fca5e6e-5a1b-45f8-838e-1fa234468a78-plugins-config-volume\") pod \"7fca5e6e-5a1b-45f8-838e-1fa234468a78\" (UID: \"7fca5e6e-5a1b-45f8-838e-1fa234468a78\") " Oct 31 03:17:16.487834 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.487698 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fca5e6e-5a1b-45f8-838e-1fa234468a78-kube-api-access-5scdm\" (UniqueName: \"kubernetes.io/projected/7fca5e6e-5a1b-45f8-838e-1fa234468a78-kube-api-access-5scdm\") pod \"7fca5e6e-5a1b-45f8-838e-1fa234468a78\" (UID: \"7fca5e6e-5a1b-45f8-838e-1fa234468a78\") " Oct 31 03:17:16.488547 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.488519 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fca5e6e-5a1b-45f8-838e-1fa234468a78-plugins-config-volume" pod "7fca5e6e-5a1b-45f8-838e-1fa234468a78" (UID: "7fca5e6e-5a1b-45f8-838e-1fa234468a78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:16.490902 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.490872 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fca5e6e-5a1b-45f8-838e-1fa234468a78-kube-api-access-5scdm" pod "7fca5e6e-5a1b-45f8-838e-1fa234468a78" (UID: "7fca5e6e-5a1b-45f8-838e-1fa234468a78"). InnerVolumeSpecName "kube-api-access-5scdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:16.590815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.590776 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fca5e6e-5a1b-45f8-838e-1fa234468a78-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:16.590815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:16.590811 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5scdm\" (UniqueName: \"kubernetes.io/projected/7fca5e6e-5a1b-45f8-838e-1fa234468a78-kube-api-access-5scdm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:17.445963 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:17.445931 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs" Oct 31 03:17:17.503368 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:17.503303 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs"] Oct 31 03:17:17.511616 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:17.511579 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ad1f0-5899cdd987-qcdhs"] Oct 31 03:17:18.438473 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:18.438446 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fca5e6e-5a1b-45f8-838e-1fa234468a78" path="/var/lib/kubelet/pods/7fca5e6e-5a1b-45f8-838e-1fa234468a78/volumes" Oct 31 03:17:19.847842 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:19.847806 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:17:20.001862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:19.997273 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-envoy\") pod \"133bc9f0-a624-473c-84a1-c2db415a52aa\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " Oct 31 03:17:20.001862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:19.997433 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-certs\") pod \"133bc9f0-a624-473c-84a1-c2db415a52aa\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " Oct 31 03:17:20.001862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:19.997641 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-data\") pod \"133bc9f0-a624-473c-84a1-c2db415a52aa\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " Oct 31 03:17:20.001862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:19.997775 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-token\" (UniqueName: \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-token\") pod \"133bc9f0-a624-473c-84a1-c2db415a52aa\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " Oct 31 03:17:20.001862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:19.997817 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-podinfo\") pod \"133bc9f0-a624-473c-84a1-c2db415a52aa\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " Oct 31 03:17:20.001862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:19.997849 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133bc9f0-a624-473c-84a1-c2db415a52aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133bc9f0-a624-473c-84a1-c2db415a52aa-istiod-ca-cert\") pod \"133bc9f0-a624-473c-84a1-c2db415a52aa\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " Oct 31 03:17:20.001862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:19.997891 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-credential-socket\") pod \"133bc9f0-a624-473c-84a1-c2db415a52aa\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " Oct 31 03:17:20.001862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:19.997933 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-kube-api-access-chcw2\" (UniqueName: \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-kube-api-access-chcw2\") pod \"133bc9f0-a624-473c-84a1-c2db415a52aa\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " Oct 31 03:17:20.001862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:19.997986 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-socket\") pod \"133bc9f0-a624-473c-84a1-c2db415a52aa\" (UID: \"133bc9f0-a624-473c-84a1-c2db415a52aa\") " Oct 31 03:17:20.004202 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.003657 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-socket" pod "133bc9f0-a624-473c-84a1-c2db415a52aa" (UID: "133bc9f0-a624-473c-84a1-c2db415a52aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:20.004553 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.004517 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-credential-socket" pod "133bc9f0-a624-473c-84a1-c2db415a52aa" (UID: "133bc9f0-a624-473c-84a1-c2db415a52aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:20.004847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.004740 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-certs" pod "133bc9f0-a624-473c-84a1-c2db415a52aa" (UID: "133bc9f0-a624-473c-84a1-c2db415a52aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:20.005182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.005146 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-data" pod "133bc9f0-a624-473c-84a1-c2db415a52aa" (UID: "133bc9f0-a624-473c-84a1-c2db415a52aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:20.005182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.005175 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133bc9f0-a624-473c-84a1-c2db415a52aa-istiod-ca-cert" pod "133bc9f0-a624-473c-84a1-c2db415a52aa" (UID: "133bc9f0-a624-473c-84a1-c2db415a52aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:20.007102 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.007022 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-token" pod "133bc9f0-a624-473c-84a1-c2db415a52aa" (UID: "133bc9f0-a624-473c-84a1-c2db415a52aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:20.007247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.007093 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-envoy" pod "133bc9f0-a624-473c-84a1-c2db415a52aa" (UID: "133bc9f0-a624-473c-84a1-c2db415a52aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:20.007247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.007168 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-podinfo" pod "133bc9f0-a624-473c-84a1-c2db415a52aa" (UID: "133bc9f0-a624-473c-84a1-c2db415a52aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:20.007383 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.007353 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-kube-api-access-chcw2" pod "133bc9f0-a624-473c-84a1-c2db415a52aa" (UID: "133bc9f0-a624-473c-84a1-c2db415a52aa"). InnerVolumeSpecName "kube-api-access-chcw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:20.100686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.100599 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:20.100686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.100630 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:20.100686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.100641 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133bc9f0-a624-473c-84a1-c2db415a52aa-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:20.100686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.100649 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:20.100686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.100658 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chcw2\" (UniqueName: \"kubernetes.io/projected/133bc9f0-a624-473c-84a1-c2db415a52aa-kube-api-access-chcw2\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:20.100686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.100669 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:20.100686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.100682 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:20.101101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.100692 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:20.101101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.100703 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133bc9f0-a624-473c-84a1-c2db415a52aa-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:20.481840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.481803 4456 generic.go:356] "Generic (PLEG): container finished" podID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerID="4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a" exitCode=0 Oct 31 03:17:20.481998 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.481865 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" Oct 31 03:17:20.481998 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.481886 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" event={"ID":"133bc9f0-a624-473c-84a1-c2db415a52aa","Type":"ContainerDied","Data":"4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a"} Oct 31 03:17:20.481998 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.481955 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x" event={"ID":"133bc9f0-a624-473c-84a1-c2db415a52aa","Type":"ContainerDied","Data":"9937659e80cad8671e2c0e4958746175df0309bf8a372d7e0257743255ae9033"} Oct 31 03:17:20.481998 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.481978 4456 scope.go:122] "RemoveContainer" containerID="4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a" Oct 31 03:17:20.495262 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.495240 4456 scope.go:122] "RemoveContainer" containerID="4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a" Oct 31 03:17:20.495608 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:17:20.495588 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a\": container with ID starting with 4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a not found: ID does not exist" containerID="4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a" Oct 31 03:17:20.495703 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.495619 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a"} err="failed to get container status \"4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a\": rpc error: code = NotFound desc = could not find container \"4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a\": container with ID starting with 4dae609ffc84bce68cafe3bb38a3775fe9dc357f8f848cb464b1270b9e71ab1a not found: ID does not exist" Oct 31 03:17:20.514602 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.514571 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x"] Oct 31 03:17:20.519901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:20.519877 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gzw4x"] Oct 31 03:17:22.438912 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:22.438883 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" path="/var/lib/kubelet/pods/133bc9f0-a624-473c-84a1-c2db415a52aa/volumes" Oct 31 03:17:25.744894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:25.744860 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs"] Oct 31 03:17:25.749495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:25.749468 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-tzgcs"] Oct 31 03:17:26.439215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.439175 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa24c702-323b-41a0-a449-4682a705a5d6" path="/var/lib/kubelet/pods/aa24c702-323b-41a0-a449-4682a705a5d6/volumes" Oct 31 03:17:26.738708 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.738614 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8"] Oct 31 03:17:26.739532 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739508 4456 cpu_manager.go:414] "containerMap: removing container" podUID="5c62f1ec-97fb-4f46-991c-9088c99e11aa" containerName="revoke-keys" Oct 31 03:17:26.739532 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739531 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c62f1ec-97fb-4f46-991c-9088c99e11aa" containerName="revoke-keys" Oct 31 03:17:26.739720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739569 4456 cpu_manager.go:414] "containerMap: removing container" podUID="18c29c9c-1fd0-47c9-bd79-0bbffb9e3902" containerName="payload-processing" Oct 31 03:17:26.739720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739576 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c29c9c-1fd0-47c9-bd79-0bbffb9e3902" containerName="payload-processing" Oct 31 03:17:26.739720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739594 4456 cpu_manager.go:414] "containerMap: removing container" podUID="7fca5e6e-5a1b-45f8-838e-1fa234468a78" containerName="payload-pre-processing" Oct 31 03:17:26.739720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739602 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fca5e6e-5a1b-45f8-838e-1fa234468a78" containerName="payload-pre-processing" Oct 31 03:17:26.739720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739614 4456 cpu_manager.go:414] "containerMap: removing container" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" Oct 31 03:17:26.739720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739622 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" Oct 31 03:17:26.739720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739668 4456 cpu_manager.go:414] "containerMap: removing container" podUID="6943cde0-73dd-4c09-b335-531048357951" containerName="payload-pre-processing" Oct 31 03:17:26.739720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739677 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6943cde0-73dd-4c09-b335-531048357951" containerName="payload-pre-processing" Oct 31 03:17:26.739720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739711 4456 cpu_manager.go:414] "containerMap: removing container" podUID="5ad03736-abe0-4140-bcea-5b915b72269b" containerName="revoke-keys" Oct 31 03:17:26.739720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739718 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad03736-abe0-4140-bcea-5b915b72269b" containerName="revoke-keys" Oct 31 03:17:26.740235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739895 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6943cde0-73dd-4c09-b335-531048357951" containerName="payload-pre-processing" Oct 31 03:17:26.740235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739921 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad03736-abe0-4140-bcea-5b915b72269b" containerName="revoke-keys" Oct 31 03:17:26.740235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739934 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c62f1ec-97fb-4f46-991c-9088c99e11aa" containerName="revoke-keys" Oct 31 03:17:26.740235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739949 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c29c9c-1fd0-47c9-bd79-0bbffb9e3902" containerName="payload-processing" Oct 31 03:17:26.740235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739961 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fca5e6e-5a1b-45f8-838e-1fa234468a78" containerName="payload-pre-processing" Oct 31 03:17:26.740235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.739972 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="133bc9f0-a624-473c-84a1-c2db415a52aa" containerName="istio-proxy" Oct 31 03:17:26.758062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.752273 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:26.762059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.759325 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:17:26.762059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.759919 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:17:26.784968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.784930 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8"] Oct 31 03:17:26.908060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.906082 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:26.908060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.906190 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ssbb\" (UniqueName: \"kubernetes.io/projected/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-kube-api-access-4ssbb\") pod \"maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:26.908060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:26.906445 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:27.011868 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:27.011762 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:27.011868 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:27.011832 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ssbb\" (UniqueName: \"kubernetes.io/projected/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-kube-api-access-4ssbb\") pod \"maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:27.012166 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:27.012020 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:27.014330 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:27.014298 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:27.031086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:27.029245 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ssbb\" (UniqueName: \"kubernetes.io/projected/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-kube-api-access-4ssbb\") pod \"maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:27.031086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:27.029419 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:27.105909 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:27.105876 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:27.413020 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:17:27.412989 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b5c830_aacb_4b6f_9e1e_f4ac1393731a.slice/crio-a5da2330187372aeb346c4729d43d62cb0e708bb43479a477bc1f10817d2ff05 WatchSource:0}: Error finding container a5da2330187372aeb346c4729d43d62cb0e708bb43479a477bc1f10817d2ff05: Status 404 returned error can't find the container with id a5da2330187372aeb346c4729d43d62cb0e708bb43479a477bc1f10817d2ff05 Oct 31 03:17:27.414471 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:27.414445 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8"] Oct 31 03:17:27.622825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:27.622776 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" event={"ID":"33b5c830-aacb-4b6f-9e1e-f4ac1393731a","Type":"ContainerStarted","Data":"a5da2330187372aeb346c4729d43d62cb0e708bb43479a477bc1f10817d2ff05"} Oct 31 03:17:28.633844 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:28.633803 4456 generic.go:356] "Generic (PLEG): container finished" podID="33b5c830-aacb-4b6f-9e1e-f4ac1393731a" containerID="06522683fad1101988bc7f0af015ad25310b4068a3aa379127dc7585b7507a78" exitCode=0 Oct 31 03:17:28.634318 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:28.633846 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" event={"ID":"33b5c830-aacb-4b6f-9e1e-f4ac1393731a","Type":"ContainerDied","Data":"06522683fad1101988bc7f0af015ad25310b4068a3aa379127dc7585b7507a78"} Oct 31 03:17:29.903658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:29.903629 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:29.996388 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:29.996350 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-kube-api-access-4ssbb\" (UniqueName: \"kubernetes.io/projected/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-kube-api-access-4ssbb\") pod \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " Oct 31 03:17:29.996583 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:29.996565 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-openshift-ca\") pod \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " Oct 31 03:17:29.996634 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:29.996616 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-xks-ca\") pod \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\" (UID: \"33b5c830-aacb-4b6f-9e1e-f4ac1393731a\") " Oct 31 03:17:29.997004 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:29.996946 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-openshift-ca" pod "33b5c830-aacb-4b6f-9e1e-f4ac1393731a" (UID: "33b5c830-aacb-4b6f-9e1e-f4ac1393731a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:29.997619 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:29.997595 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:29.999363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:29.999332 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-kube-api-access-4ssbb" pod "33b5c830-aacb-4b6f-9e1e-f4ac1393731a" (UID: "33b5c830-aacb-4b6f-9e1e-f4ac1393731a"). InnerVolumeSpecName "kube-api-access-4ssbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:29.999460 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:29.999409 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-xks-ca" pod "33b5c830-aacb-4b6f-9e1e-f4ac1393731a" (UID: "33b5c830-aacb-4b6f-9e1e-f4ac1393731a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:30.100658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:30.100590 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ssbb\" (UniqueName: \"kubernetes.io/projected/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-kube-api-access-4ssbb\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:30.100658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:30.100631 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b5c830-aacb-4b6f-9e1e-f4ac1393731a-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:30.708126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:30.708076 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" event={"ID":"33b5c830-aacb-4b6f-9e1e-f4ac1393731a","Type":"ContainerDied","Data":"a5da2330187372aeb346c4729d43d62cb0e708bb43479a477bc1f10817d2ff05"} Oct 31 03:17:30.708449 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:30.708401 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5da2330187372aeb346c4729d43d62cb0e708bb43479a477bc1f10817d2ff05" Oct 31 03:17:30.708669 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:30.708653 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8" Oct 31 03:17:31.470605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:31.470563 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm"] Oct 31 03:17:31.476739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:31.476709 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb"] Oct 31 03:17:32.136430 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:17:32.136392 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:17:32.940150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.940107 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh"] Oct 31 03:17:32.941441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.941414 4456 cpu_manager.go:414] "containerMap: removing container" podUID="33b5c830-aacb-4b6f-9e1e-f4ac1393731a" containerName="revoke-keys" Oct 31 03:17:32.941441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.941441 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b5c830-aacb-4b6f-9e1e-f4ac1393731a" containerName="revoke-keys" Oct 31 03:17:32.941668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.941654 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b5c830-aacb-4b6f-9e1e-f4ac1393731a" containerName="revoke-keys" Oct 31 03:17:32.948970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.948946 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh"] Oct 31 03:17:32.949153 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.949114 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:32.953717 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.953692 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:17:32.955666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.955647 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:17:32.966979 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.966944 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvr46\" (UniqueName: \"kubernetes.io/projected/cdca02cb-b166-419f-adeb-dc8c101c7a19-kube-api-access-tvr46\") pod \"maas-api-revoke-keys-e2e-mt-750c955f-lm6vh\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:32.967179 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.967154 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-750c955f-lm6vh\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:32.967492 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:32.967458 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-750c955f-lm6vh\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:33.069543 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:33.069509 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-750c955f-lm6vh\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:33.069724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:33.069695 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-750c955f-lm6vh\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:33.069800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:33.069785 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvr46\" (UniqueName: \"kubernetes.io/projected/cdca02cb-b166-419f-adeb-dc8c101c7a19-kube-api-access-tvr46\") pod \"maas-api-revoke-keys-e2e-mt-750c955f-lm6vh\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:33.070397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:33.070365 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-750c955f-lm6vh\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:33.072290 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:33.072263 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-750c955f-lm6vh\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:33.082970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:33.082946 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvr46\" (UniqueName: \"kubernetes.io/projected/cdca02cb-b166-419f-adeb-dc8c101c7a19-kube-api-access-tvr46\") pod \"maas-api-revoke-keys-e2e-mt-750c955f-lm6vh\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:33.270623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:33.270537 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:33.568764 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:17:33.568730 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdca02cb_b166_419f_adeb_dc8c101c7a19.slice/crio-ebd756f77cd6611b8ff6e0ec750fdd76f73ff7637982376925ffba6f0b4a19e9 WatchSource:0}: Error finding container ebd756f77cd6611b8ff6e0ec750fdd76f73ff7637982376925ffba6f0b4a19e9: Status 404 returned error can't find the container with id ebd756f77cd6611b8ff6e0ec750fdd76f73ff7637982376925ffba6f0b4a19e9 Oct 31 03:17:33.570445 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:33.570269 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh"] Oct 31 03:17:33.796655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:33.796611 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" event={"ID":"cdca02cb-b166-419f-adeb-dc8c101c7a19","Type":"ContainerStarted","Data":"ebd756f77cd6611b8ff6e0ec750fdd76f73ff7637982376925ffba6f0b4a19e9"} Oct 31 03:17:34.808827 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:34.808789 4456 generic.go:356] "Generic (PLEG): container finished" podID="cdca02cb-b166-419f-adeb-dc8c101c7a19" containerID="6f9698c509eb86099670284bcb46a704fec4c1f25ebc17cc57ef47c50a30e495" exitCode=0 Oct 31 03:17:34.809214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:34.808879 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" event={"ID":"cdca02cb-b166-419f-adeb-dc8c101c7a19","Type":"ContainerDied","Data":"6f9698c509eb86099670284bcb46a704fec4c1f25ebc17cc57ef47c50a30e495"} Oct 31 03:17:36.012565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.012536 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:36.066121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.066088 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-xks-ca\") pod \"cdca02cb-b166-419f-adeb-dc8c101c7a19\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " Oct 31 03:17:36.066307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.066200 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdca02cb-b166-419f-adeb-dc8c101c7a19-kube-api-access-tvr46\" (UniqueName: \"kubernetes.io/projected/cdca02cb-b166-419f-adeb-dc8c101c7a19-kube-api-access-tvr46\") pod \"cdca02cb-b166-419f-adeb-dc8c101c7a19\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " Oct 31 03:17:36.066396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.066380 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-openshift-ca\") pod \"cdca02cb-b166-419f-adeb-dc8c101c7a19\" (UID: \"cdca02cb-b166-419f-adeb-dc8c101c7a19\") " Oct 31 03:17:36.066819 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.066794 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-openshift-ca" pod "cdca02cb-b166-419f-adeb-dc8c101c7a19" (UID: "cdca02cb-b166-419f-adeb-dc8c101c7a19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:36.067266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.067241 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:36.068846 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.068818 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-xks-ca" pod "cdca02cb-b166-419f-adeb-dc8c101c7a19" (UID: "cdca02cb-b166-419f-adeb-dc8c101c7a19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:36.068994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.068976 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdca02cb-b166-419f-adeb-dc8c101c7a19-kube-api-access-tvr46" pod "cdca02cb-b166-419f-adeb-dc8c101c7a19" (UID: "cdca02cb-b166-419f-adeb-dc8c101c7a19"). InnerVolumeSpecName "kube-api-access-tvr46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:36.169424 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.169376 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdca02cb-b166-419f-adeb-dc8c101c7a19-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:36.169424 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.169414 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvr46\" (UniqueName: \"kubernetes.io/projected/cdca02cb-b166-419f-adeb-dc8c101c7a19-kube-api-access-tvr46\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:36.831952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.831804 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" Oct 31 03:17:36.831952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.831841 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh" event={"ID":"cdca02cb-b166-419f-adeb-dc8c101c7a19","Type":"ContainerDied","Data":"ebd756f77cd6611b8ff6e0ec750fdd76f73ff7637982376925ffba6f0b4a19e9"} Oct 31 03:17:36.831952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:36.831900 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd756f77cd6611b8ff6e0ec750fdd76f73ff7637982376925ffba6f0b4a19e9" Oct 31 03:17:43.114606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:43.114569 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm"] Oct 31 03:17:45.636300 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:45.636251 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx"] Oct 31 03:17:45.642519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:45.642486 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad1f0-bx5zx"] Oct 31 03:17:46.558059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:46.552584 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" containerID="cri-o://818fbff4aae0dcceb2238d6a349b2e246c63d28a7ce22bfdec443d4e5c299d2f" gracePeriod=45 Oct 31 03:17:46.558059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:46.552725 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" containerID="cri-o://d28822ff6f0e189d0f41ea0ff8acd7748ae3c4c29bd5941e088e236c963081cb" gracePeriod=45 Oct 31 03:17:46.693070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:46.690008 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad03736-abe0-4140-bcea-5b915b72269b" path="/var/lib/kubelet/pods/5ad03736-abe0-4140-bcea-5b915b72269b/volumes" Oct 31 03:17:46.825566 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:46.824092 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": dial tcp 10.133.0.72:8000: connect: connection refused" Oct 31 03:17:46.853052 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:46.845837 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": EOF" Oct 31 03:17:47.097566 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.097477 4456 generic.go:356] "Generic (PLEG): container finished" podID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerID="818fbff4aae0dcceb2238d6a349b2e246c63d28a7ce22bfdec443d4e5c299d2f" exitCode=0 Oct 31 03:17:47.098408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.098235 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" event={"ID":"a0a17bda-c051-44fb-bc97-1b3aa1a4c290","Type":"ContainerDied","Data":"818fbff4aae0dcceb2238d6a349b2e246c63d28a7ce22bfdec443d4e5c299d2f"} Oct 31 03:17:47.101572 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.101355 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" event={"ID":"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51","Type":"ContainerDied","Data":"d28822ff6f0e189d0f41ea0ff8acd7748ae3c4c29bd5941e088e236c963081cb"} Oct 31 03:17:47.101729 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.101152 4456 generic.go:356] "Generic (PLEG): container finished" podID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerID="d28822ff6f0e189d0f41ea0ff8acd7748ae3c4c29bd5941e088e236c963081cb" exitCode=0 Oct 31 03:17:47.177415 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.177018 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:17:47.246227 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.246187 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tmp-dir\") pod \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " Oct 31 03:17:47.246404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.246274 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-kube-api-access-m6x95\" (UniqueName: \"kubernetes.io/projected/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-kube-api-access-m6x95\") pod \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " Oct 31 03:17:47.246404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.246324 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-dshm\") pod \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " Oct 31 03:17:47.246404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.246370 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-model-cache\") pod \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " Oct 31 03:17:47.246589 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.246418 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-home\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-home\") pod \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " Oct 31 03:17:47.246657 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.246635 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tls-certs\") pod \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\" (UID: \"a0a17bda-c051-44fb-bc97-1b3aa1a4c290\") " Oct 31 03:17:47.246943 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.246909 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-model-cache" pod "a0a17bda-c051-44fb-bc97-1b3aa1a4c290" (UID: "a0a17bda-c051-44fb-bc97-1b3aa1a4c290"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.247221 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.247195 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tmp-dir" pod "a0a17bda-c051-44fb-bc97-1b3aa1a4c290" (UID: "a0a17bda-c051-44fb-bc97-1b3aa1a4c290"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.247599 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.247572 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.247702 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.247600 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.247824 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.247803 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-home" pod "a0a17bda-c051-44fb-bc97-1b3aa1a4c290" (UID: "a0a17bda-c051-44fb-bc97-1b3aa1a4c290"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.253529 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.251540 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tls-certs" pod "a0a17bda-c051-44fb-bc97-1b3aa1a4c290" (UID: "a0a17bda-c051-44fb-bc97-1b3aa1a4c290"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:47.255144 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.254748 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-kube-api-access-m6x95" pod "a0a17bda-c051-44fb-bc97-1b3aa1a4c290" (UID: "a0a17bda-c051-44fb-bc97-1b3aa1a4c290"). InnerVolumeSpecName "kube-api-access-m6x95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:47.273377 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.271772 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-dshm" pod "a0a17bda-c051-44fb-bc97-1b3aa1a4c290" (UID: "a0a17bda-c051-44fb-bc97-1b3aa1a4c290"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.394270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.394172 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6x95\" (UniqueName: \"kubernetes.io/projected/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-kube-api-access-m6x95\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.394270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.394209 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.394270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.394226 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.394270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.394239 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a17bda-c051-44fb-bc97-1b3aa1a4c290-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.439396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.439360 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": dial tcp 10.133.0.65:8000: connect: connection refused" Oct 31 03:17:47.439518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.439465 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:17:47.584137 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.577521 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:17:47.702115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.700813 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-model-cache\") pod \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " Oct 31 03:17:47.702115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.700850 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-dshm\") pod \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " Oct 31 03:17:47.702115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.700979 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-home\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-home\") pod \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " Oct 31 03:17:47.702115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.701011 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-kube-api-access-q76qm\" (UniqueName: \"kubernetes.io/projected/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-kube-api-access-q76qm\") pod \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " Oct 31 03:17:47.702115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.701079 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tmp-dir\") pod \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " Oct 31 03:17:47.702115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.701298 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-home" pod "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" (UID: "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.702115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.701566 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tmp-dir" pod "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" (UID: "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.702115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.701581 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tls-certs\") pod \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\" (UID: \"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51\") " Oct 31 03:17:47.702115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.701686 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-model-cache" pod "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" (UID: "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.703006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.702577 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.703006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.702600 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.703006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.702615 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.706145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.706115 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-kube-api-access-q76qm" pod "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" (UID: "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51"). InnerVolumeSpecName "kube-api-access-q76qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:47.706275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.706201 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-dshm" pod "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" (UID: "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.707896 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.707868 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tls-certs" pod "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" (UID: "ea1a1ec2-20e7-4119-a3a8-82a8d6142d51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:47.804792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.804757 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.804792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.804788 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.804977 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:47.804801 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q76qm\" (UniqueName: \"kubernetes.io/projected/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51-kube-api-access-q76qm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:48.115221 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.115119 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" event={"ID":"ea1a1ec2-20e7-4119-a3a8-82a8d6142d51","Type":"ContainerDied","Data":"1c92b51be7cdd32363d96b008e922e50fd65b002935507d51bafdc69ec66b57b"} Oct 31 03:17:48.115221 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.115187 4456 scope.go:122] "RemoveContainer" containerID="d28822ff6f0e189d0f41ea0ff8acd7748ae3c4c29bd5941e088e236c963081cb" Oct 31 03:17:48.115441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.115130 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb" Oct 31 03:17:48.115441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.115344 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" podUID="7c40da77-42b0-4315-bf18-d38ecda7f5ce" containerName="payload-pre-processing" containerID="cri-o://a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112" gracePeriod=25 Oct 31 03:17:48.117172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.117148 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" event={"ID":"a0a17bda-c051-44fb-bc97-1b3aa1a4c290","Type":"ContainerDied","Data":"0820a010e9f12c33ee9ad140cf16b09d6815c4cc1b7762c5f6725dd040a10fb2"} Oct 31 03:17:48.117322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.117306 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm" Oct 31 03:17:48.129710 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.129689 4456 scope.go:122] "RemoveContainer" containerID="818fbff4aae0dcceb2238d6a349b2e246c63d28a7ce22bfdec443d4e5c299d2f" Oct 31 03:17:48.223818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.223791 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm"] Oct 31 03:17:48.232252 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.232216 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-2-w5-56afbd-kserve-56f5c49985-qzgdm"] Oct 31 03:17:48.238603 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.238577 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb"] Oct 31 03:17:48.243717 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.243692 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/distinct-w5-56afbd-kserve-85dd9b46cd-clnbb"] Oct 31 03:17:48.540204 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.540166 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:17:48.541349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.541322 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" path="/var/lib/kubelet/pods/a0a17bda-c051-44fb-bc97-1b3aa1a4c290/volumes" Oct 31 03:17:48.542073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.542050 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" path="/var/lib/kubelet/pods/ea1a1ec2-20e7-4119-a3a8-82a8d6142d51/volumes" Oct 31 03:17:48.631859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.631821 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c40da77-42b0-4315-bf18-d38ecda7f5ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c40da77-42b0-4315-bf18-d38ecda7f5ce-plugins-config-volume\") pod \"7c40da77-42b0-4315-bf18-d38ecda7f5ce\" (UID: \"7c40da77-42b0-4315-bf18-d38ecda7f5ce\") " Oct 31 03:17:48.632135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.632092 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c40da77-42b0-4315-bf18-d38ecda7f5ce-kube-api-access-ptbnt\" (UniqueName: \"kubernetes.io/projected/7c40da77-42b0-4315-bf18-d38ecda7f5ce-kube-api-access-ptbnt\") pod \"7c40da77-42b0-4315-bf18-d38ecda7f5ce\" (UID: \"7c40da77-42b0-4315-bf18-d38ecda7f5ce\") " Oct 31 03:17:48.632362 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.632335 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c40da77-42b0-4315-bf18-d38ecda7f5ce-plugins-config-volume" pod "7c40da77-42b0-4315-bf18-d38ecda7f5ce" (UID: "7c40da77-42b0-4315-bf18-d38ecda7f5ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:48.633475 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.633443 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c40da77-42b0-4315-bf18-d38ecda7f5ce-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:48.635787 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.635761 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c40da77-42b0-4315-bf18-d38ecda7f5ce-kube-api-access-ptbnt" pod "7c40da77-42b0-4315-bf18-d38ecda7f5ce" (UID: "7c40da77-42b0-4315-bf18-d38ecda7f5ce"). InnerVolumeSpecName "kube-api-access-ptbnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:48.735897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:48.735819 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptbnt\" (UniqueName: \"kubernetes.io/projected/7c40da77-42b0-4315-bf18-d38ecda7f5ce-kube-api-access-ptbnt\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.127428 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:49.127339 4456 generic.go:356] "Generic (PLEG): container finished" podID="7c40da77-42b0-4315-bf18-d38ecda7f5ce" containerID="a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112" exitCode=0 Oct 31 03:17:49.127428 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:49.127401 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" Oct 31 03:17:49.127625 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:49.127421 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" event={"ID":"7c40da77-42b0-4315-bf18-d38ecda7f5ce","Type":"ContainerDied","Data":"a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112"} Oct 31 03:17:49.127625 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:49.127505 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm" event={"ID":"7c40da77-42b0-4315-bf18-d38ecda7f5ce","Type":"ContainerDied","Data":"90a2e40ffb2a31889204dcfe871ac983c042a316a70d210c9a61f3a0171aa258"} Oct 31 03:17:49.127625 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:49.127536 4456 scope.go:122] "RemoveContainer" containerID="a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112" Oct 31 03:17:49.140469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:49.140063 4456 scope.go:122] "RemoveContainer" containerID="a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112" Oct 31 03:17:49.140469 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:17:49.140386 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112\": container with ID starting with a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112 not found: ID does not exist" containerID="a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112" Oct 31 03:17:49.140469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:49.140420 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112"} err="failed to get container status \"a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112\": rpc error: code = NotFound desc = could not find container \"a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112\": container with ID starting with a148695ab51cd84c802019da640e9e5df77478097bddcc1ac959b92f03b8a112 not found: ID does not exist" Oct 31 03:17:49.159776 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:49.159744 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm"] Oct 31 03:17:49.168673 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:49.168647 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-750c955f-7574c6867-h5zwm"] Oct 31 03:17:50.449769 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:50.448317 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c40da77-42b0-4315-bf18-d38ecda7f5ce" path="/var/lib/kubelet/pods/7c40da77-42b0-4315-bf18-d38ecda7f5ce/volumes" Oct 31 03:17:51.319018 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.318978 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f"] Oct 31 03:17:51.319827 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.319802 4456 cpu_manager.go:414] "containerMap: removing container" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" Oct 31 03:17:51.319827 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.319820 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" Oct 31 03:17:51.319994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.319853 4456 cpu_manager.go:414] "containerMap: removing container" podUID="cdca02cb-b166-419f-adeb-dc8c101c7a19" containerName="revoke-keys" Oct 31 03:17:51.319994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.319858 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdca02cb-b166-419f-adeb-dc8c101c7a19" containerName="revoke-keys" Oct 31 03:17:51.319994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.319883 4456 cpu_manager.go:414] "containerMap: removing container" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" Oct 31 03:17:51.319994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.319888 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" Oct 31 03:17:51.319994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.319928 4456 cpu_manager.go:414] "containerMap: removing container" podUID="7c40da77-42b0-4315-bf18-d38ecda7f5ce" containerName="payload-pre-processing" Oct 31 03:17:51.319994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.319934 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c40da77-42b0-4315-bf18-d38ecda7f5ce" containerName="payload-pre-processing" Oct 31 03:17:51.320187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.320056 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a17bda-c051-44fb-bc97-1b3aa1a4c290" containerName="main" Oct 31 03:17:51.320187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.320070 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c40da77-42b0-4315-bf18-d38ecda7f5ce" containerName="payload-pre-processing" Oct 31 03:17:51.320187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.320084 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1a1ec2-20e7-4119-a3a8-82a8d6142d51" containerName="main" Oct 31 03:17:51.320187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.320098 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdca02cb-b166-419f-adeb-dc8c101c7a19" containerName="revoke-keys" Oct 31 03:17:51.325173 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.325155 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:51.331627 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.331606 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:17:51.331733 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.331704 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:17:51.333393 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.333371 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f"] Oct 31 03:17:51.439789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.439755 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfrk6\" (UniqueName: \"kubernetes.io/projected/257a8229-2fdf-4467-9360-b0d6a822dfb1-kube-api-access-qfrk6\") pod \"maas-api-revoke-sub-33cf5ce7-gm29f\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:51.440000 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.439984 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cf5ce7-gm29f\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:51.440060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.440015 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cf5ce7-gm29f\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:51.544794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.544760 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cf5ce7-gm29f\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:51.545506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.544806 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cf5ce7-gm29f\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:51.545506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.545132 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfrk6\" (UniqueName: \"kubernetes.io/projected/257a8229-2fdf-4467-9360-b0d6a822dfb1-kube-api-access-qfrk6\") pod \"maas-api-revoke-sub-33cf5ce7-gm29f\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:51.545961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.545929 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cf5ce7-gm29f\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:51.550699 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.550674 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cf5ce7-gm29f\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:51.561120 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.561089 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfrk6\" (UniqueName: \"kubernetes.io/projected/257a8229-2fdf-4467-9360-b0d6a822dfb1-kube-api-access-qfrk6\") pod \"maas-api-revoke-sub-33cf5ce7-gm29f\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:51.672060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:51.668436 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:52.211239 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:17:52.211197 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257a8229_2fdf_4467_9360_b0d6a822dfb1.slice/crio-15ea5507c8499f040e6e3dcbc55d6e9f77c5fd46f141dc98f891213bd0d05f11 WatchSource:0}: Error finding container 15ea5507c8499f040e6e3dcbc55d6e9f77c5fd46f141dc98f891213bd0d05f11: Status 404 returned error can't find the container with id 15ea5507c8499f040e6e3dcbc55d6e9f77c5fd46f141dc98f891213bd0d05f11 Oct 31 03:17:52.212523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:52.212497 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f"] Oct 31 03:17:53.227125 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:53.226937 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" event={"ID":"257a8229-2fdf-4467-9360-b0d6a822dfb1","Type":"ContainerStarted","Data":"a5f26be4a85941dbf689c835f919cac6a26f88685219e1abb1cbedb7fa2d0a04"} Oct 31 03:17:53.227125 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:53.227070 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" event={"ID":"257a8229-2fdf-4467-9360-b0d6a822dfb1","Type":"ContainerStarted","Data":"15ea5507c8499f040e6e3dcbc55d6e9f77c5fd46f141dc98f891213bd0d05f11"} Oct 31 03:17:54.209061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.197564 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" podStartSLOduration=3.197544606 podStartE2EDuration="3.197544606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:53.259242544 +0000 UTC m=+1354.939768685" watchObservedRunningTime="2026-10-31 03:17:54.197544606 +0000 UTC m=+1355.878070734" Oct 31 03:17:54.209061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.206273 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59"] Oct 31 03:17:54.258805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.257254 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:54.262119 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.260641 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59"] Oct 31 03:17:54.336327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.335946 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602b57f0-xxr59\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:54.336327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.336007 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ctjr\" (UniqueName: \"kubernetes.io/projected/f5287f4d-6337-4712-b9b9-bcc96d614cee-kube-api-access-2ctjr\") pod \"maas-api-revoke-sub-602b57f0-xxr59\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:54.336327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.336063 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602b57f0-xxr59\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:54.366192 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.366157 4456 generic.go:356] "Generic (PLEG): container finished" podID="257a8229-2fdf-4467-9360-b0d6a822dfb1" containerID="a5f26be4a85941dbf689c835f919cac6a26f88685219e1abb1cbedb7fa2d0a04" exitCode=0 Oct 31 03:17:54.366785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.366425 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" event={"ID":"257a8229-2fdf-4467-9360-b0d6a822dfb1","Type":"ContainerDied","Data":"a5f26be4a85941dbf689c835f919cac6a26f88685219e1abb1cbedb7fa2d0a04"} Oct 31 03:17:54.437361 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.437328 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602b57f0-xxr59\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:54.437539 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.437375 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ctjr\" (UniqueName: \"kubernetes.io/projected/f5287f4d-6337-4712-b9b9-bcc96d614cee-kube-api-access-2ctjr\") pod \"maas-api-revoke-sub-602b57f0-xxr59\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:54.437539 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.437424 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602b57f0-xxr59\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:54.438373 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.438341 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602b57f0-xxr59\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:54.440233 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.440206 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602b57f0-xxr59\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:54.450702 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.450676 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ctjr\" (UniqueName: \"kubernetes.io/projected/f5287f4d-6337-4712-b9b9-bcc96d614cee-kube-api-access-2ctjr\") pod \"maas-api-revoke-sub-602b57f0-xxr59\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:54.670093 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:54.667879 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:55.056325 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:17:55.056283 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5287f4d_6337_4712_b9b9_bcc96d614cee.slice/crio-b952956b163e8daee8f55dd217a59fc809a3c81960772111a46b7bf9ba1211fa WatchSource:0}: Error finding container b952956b163e8daee8f55dd217a59fc809a3c81960772111a46b7bf9ba1211fa: Status 404 returned error can't find the container with id b952956b163e8daee8f55dd217a59fc809a3c81960772111a46b7bf9ba1211fa Oct 31 03:17:55.056503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:55.056484 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59"] Oct 31 03:17:55.450060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:55.446671 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" event={"ID":"f5287f4d-6337-4712-b9b9-bcc96d614cee","Type":"ContainerStarted","Data":"b952956b163e8daee8f55dd217a59fc809a3c81960772111a46b7bf9ba1211fa"} Oct 31 03:17:55.950288 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:55.950222 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:55.958152 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:55.958104 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-openshift-ca\") pod \"257a8229-2fdf-4467-9360-b0d6a822dfb1\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " Oct 31 03:17:55.958341 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:55.958223 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-xks-ca\") pod \"257a8229-2fdf-4467-9360-b0d6a822dfb1\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " Oct 31 03:17:55.958422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:55.958359 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257a8229-2fdf-4467-9360-b0d6a822dfb1-kube-api-access-qfrk6\" (UniqueName: \"kubernetes.io/projected/257a8229-2fdf-4467-9360-b0d6a822dfb1-kube-api-access-qfrk6\") pod \"257a8229-2fdf-4467-9360-b0d6a822dfb1\" (UID: \"257a8229-2fdf-4467-9360-b0d6a822dfb1\") " Oct 31 03:17:55.959237 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:55.959199 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-openshift-ca" pod "257a8229-2fdf-4467-9360-b0d6a822dfb1" (UID: "257a8229-2fdf-4467-9360-b0d6a822dfb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:55.961262 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:55.961232 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-xks-ca" pod "257a8229-2fdf-4467-9360-b0d6a822dfb1" (UID: "257a8229-2fdf-4467-9360-b0d6a822dfb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:55.963638 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:55.963606 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257a8229-2fdf-4467-9360-b0d6a822dfb1-kube-api-access-qfrk6" pod "257a8229-2fdf-4467-9360-b0d6a822dfb1" (UID: "257a8229-2fdf-4467-9360-b0d6a822dfb1"). InnerVolumeSpecName "kube-api-access-qfrk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:56.060859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:56.060820 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfrk6\" (UniqueName: \"kubernetes.io/projected/257a8229-2fdf-4467-9360-b0d6a822dfb1-kube-api-access-qfrk6\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:56.061101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:56.061082 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:56.061208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:56.061194 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257a8229-2fdf-4467-9360-b0d6a822dfb1-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:56.499091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:56.499002 4456 generic.go:356] "Generic (PLEG): container finished" podID="f5287f4d-6337-4712-b9b9-bcc96d614cee" containerID="c43d07c9036a09517a20ffd4c97ee00bf835716125f843f0e356f52c0785773e" exitCode=0 Oct 31 03:17:56.499739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:56.499710 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" event={"ID":"f5287f4d-6337-4712-b9b9-bcc96d614cee","Type":"ContainerDied","Data":"c43d07c9036a09517a20ffd4c97ee00bf835716125f843f0e356f52c0785773e"} Oct 31 03:17:56.502089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:56.502024 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" Oct 31 03:17:56.502200 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:56.502065 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f" event={"ID":"257a8229-2fdf-4467-9360-b0d6a822dfb1","Type":"ContainerDied","Data":"15ea5507c8499f040e6e3dcbc55d6e9f77c5fd46f141dc98f891213bd0d05f11"} Oct 31 03:17:56.502261 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:56.502198 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15ea5507c8499f040e6e3dcbc55d6e9f77c5fd46f141dc98f891213bd0d05f11" Oct 31 03:17:57.699097 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:57.699069 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:57.792493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:57.792393 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-openshift-ca\") pod \"f5287f4d-6337-4712-b9b9-bcc96d614cee\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " Oct 31 03:17:57.792675 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:57.792540 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-xks-ca\") pod \"f5287f4d-6337-4712-b9b9-bcc96d614cee\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " Oct 31 03:17:57.792742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:57.792702 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5287f4d-6337-4712-b9b9-bcc96d614cee-kube-api-access-2ctjr\" (UniqueName: \"kubernetes.io/projected/f5287f4d-6337-4712-b9b9-bcc96d614cee-kube-api-access-2ctjr\") pod \"f5287f4d-6337-4712-b9b9-bcc96d614cee\" (UID: \"f5287f4d-6337-4712-b9b9-bcc96d614cee\") " Oct 31 03:17:57.793089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:57.793020 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-openshift-ca" pod "f5287f4d-6337-4712-b9b9-bcc96d614cee" (UID: "f5287f4d-6337-4712-b9b9-bcc96d614cee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:57.794179 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:57.794158 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:57.795919 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:57.795883 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-xks-ca" pod "f5287f4d-6337-4712-b9b9-bcc96d614cee" (UID: "f5287f4d-6337-4712-b9b9-bcc96d614cee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:57.796046 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:57.795987 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5287f4d-6337-4712-b9b9-bcc96d614cee-kube-api-access-2ctjr" pod "f5287f4d-6337-4712-b9b9-bcc96d614cee" (UID: "f5287f4d-6337-4712-b9b9-bcc96d614cee"). InnerVolumeSpecName "kube-api-access-2ctjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:57.896374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:57.896333 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ctjr\" (UniqueName: \"kubernetes.io/projected/f5287f4d-6337-4712-b9b9-bcc96d614cee-kube-api-access-2ctjr\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:57.896374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:57.896374 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5287f4d-6337-4712-b9b9-bcc96d614cee-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:58.284715 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:58.282207 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh"] Oct 31 03:17:58.299059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:58.293752 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-750c955f-lm6vh"] Oct 31 03:17:58.478910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:58.477996 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdca02cb-b166-419f-adeb-dc8c101c7a19" path="/var/lib/kubelet/pods/cdca02cb-b166-419f-adeb-dc8c101c7a19/volumes" Oct 31 03:17:58.526280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:58.526254 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" Oct 31 03:17:58.526459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:58.526290 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59" event={"ID":"f5287f4d-6337-4712-b9b9-bcc96d614cee","Type":"ContainerDied","Data":"b952956b163e8daee8f55dd217a59fc809a3c81960772111a46b7bf9ba1211fa"} Oct 31 03:17:58.526459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:58.526335 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b952956b163e8daee8f55dd217a59fc809a3c81960772111a46b7bf9ba1211fa" Oct 31 03:17:58.992526 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:58.992490 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7"] Oct 31 03:17:58.992929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:17:58.992685 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerName="istio-proxy" containerID="cri-o://f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf" gracePeriod=30 Oct 31 03:18:02.183073 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:18:02.183016 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:18:04.286823 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.286794 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:18:04.350225 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.350189 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-envoy\") pod \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " Oct 31 03:18:04.350427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.350263 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-kube-api-access-5r9t9\" (UniqueName: \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-kube-api-access-5r9t9\") pod \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " Oct 31 03:18:04.350427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.350305 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-certs\") pod \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " Oct 31 03:18:04.350427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.350412 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istiod-ca-cert\") pod \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " Oct 31 03:18:04.350602 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.350481 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-socket\") pod \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " Oct 31 03:18:04.350602 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.350593 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-data\") pod \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " Oct 31 03:18:04.352379 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.350727 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-podinfo\") pod \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " Oct 31 03:18:04.352379 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.350812 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-credential-socket\") pod \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " Oct 31 03:18:04.352965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.352805 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-token\" (UniqueName: \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-token\") pod \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\" (UID: \"74a06c62-e2ef-4e7d-8763-b7bf9111d163\") " Oct 31 03:18:04.353923 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.352805 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-certs" pod "74a06c62-e2ef-4e7d-8763-b7bf9111d163" (UID: "74a06c62-e2ef-4e7d-8763-b7bf9111d163"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:04.353923 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.353809 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-credential-socket" pod "74a06c62-e2ef-4e7d-8763-b7bf9111d163" (UID: "74a06c62-e2ef-4e7d-8763-b7bf9111d163"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:04.353923 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.353129 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-data" pod "74a06c62-e2ef-4e7d-8763-b7bf9111d163" (UID: "74a06c62-e2ef-4e7d-8763-b7bf9111d163"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:04.353923 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.353416 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istiod-ca-cert" pod "74a06c62-e2ef-4e7d-8763-b7bf9111d163" (UID: "74a06c62-e2ef-4e7d-8763-b7bf9111d163"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:04.353923 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.353636 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-socket" pod "74a06c62-e2ef-4e7d-8763-b7bf9111d163" (UID: "74a06c62-e2ef-4e7d-8763-b7bf9111d163"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:04.363149 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.361433 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-podinfo" pod "74a06c62-e2ef-4e7d-8763-b7bf9111d163" (UID: "74a06c62-e2ef-4e7d-8763-b7bf9111d163"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:04.363149 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.361587 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-kube-api-access-5r9t9" pod "74a06c62-e2ef-4e7d-8763-b7bf9111d163" (UID: "74a06c62-e2ef-4e7d-8763-b7bf9111d163"). InnerVolumeSpecName "kube-api-access-5r9t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:04.363149 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.361685 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-envoy" pod "74a06c62-e2ef-4e7d-8763-b7bf9111d163" (UID: "74a06c62-e2ef-4e7d-8763-b7bf9111d163"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:04.376095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.363849 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-token" pod "74a06c62-e2ef-4e7d-8763-b7bf9111d163" (UID: "74a06c62-e2ef-4e7d-8763-b7bf9111d163"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:04.455246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.455206 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:04.455246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.455243 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:04.455487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.455257 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:04.455487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.455270 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:04.455487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.455285 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r9t9\" (UniqueName: \"kubernetes.io/projected/74a06c62-e2ef-4e7d-8763-b7bf9111d163-kube-api-access-5r9t9\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:04.455487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.455299 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:04.455487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.455311 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:04.455487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.455324 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:04.455487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.455339 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a06c62-e2ef-4e7d-8763-b7bf9111d163-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:04.589580 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.589499 4456 generic.go:356] "Generic (PLEG): container finished" podID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerID="f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf" exitCode=0 Oct 31 03:18:04.589580 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.589549 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" Oct 31 03:18:04.589759 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.589572 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" event={"ID":"74a06c62-e2ef-4e7d-8763-b7bf9111d163","Type":"ContainerDied","Data":"f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf"} Oct 31 03:18:04.589759 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.589631 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7" event={"ID":"74a06c62-e2ef-4e7d-8763-b7bf9111d163","Type":"ContainerDied","Data":"269b35cb1fde1e7ee8ea2baed2023fec21a6a02f6e41ae5a48abda672eb93261"} Oct 31 03:18:04.589759 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.589662 4456 scope.go:122] "RemoveContainer" containerID="f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf" Oct 31 03:18:04.601415 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.601392 4456 scope.go:122] "RemoveContainer" containerID="f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf" Oct 31 03:18:04.601726 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:18:04.601707 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf\": container with ID starting with f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf not found: ID does not exist" containerID="f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf" Oct 31 03:18:04.601781 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.601736 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf"} err="failed to get container status \"f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf\": rpc error: code = NotFound desc = could not find container \"f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf\": container with ID starting with f5331ec9d3f7dee68289ca7ee3aee095dc7dee1296a0b2357d8f3deede262ebf not found: ID does not exist" Oct 31 03:18:04.614874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.614848 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7"] Oct 31 03:18:04.621415 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:04.621391 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-sl6z7"] Oct 31 03:18:06.437952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:06.437920 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" path="/var/lib/kubelet/pods/74a06c62-e2ef-4e7d-8763-b7bf9111d163/volumes" Oct 31 03:18:06.556128 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:06.556084 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx"] Oct 31 03:18:07.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.386860 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6980-caa51b"] Oct 31 03:18:07.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.390193 4456 cpu_manager.go:414] "containerMap: removing container" podUID="257a8229-2fdf-4467-9360-b0d6a822dfb1" containerName="revoke-keys" Oct 31 03:18:07.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.390219 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257a8229-2fdf-4467-9360-b0d6a822dfb1" containerName="revoke-keys" Oct 31 03:18:07.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.390421 4456 cpu_manager.go:414] "containerMap: removing container" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerName="istio-proxy" Oct 31 03:18:07.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.390432 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerName="istio-proxy" Oct 31 03:18:07.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.390464 4456 cpu_manager.go:414] "containerMap: removing container" podUID="f5287f4d-6337-4712-b9b9-bcc96d614cee" containerName="revoke-keys" Oct 31 03:18:07.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.390471 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5287f4d-6337-4712-b9b9-bcc96d614cee" containerName="revoke-keys" Oct 31 03:18:07.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.390631 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5287f4d-6337-4712-b9b9-bcc96d614cee" containerName="revoke-keys" Oct 31 03:18:07.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.390642 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="257a8229-2fdf-4467-9360-b0d6a822dfb1" containerName="revoke-keys" Oct 31 03:18:07.392060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.390664 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a06c62-e2ef-4e7d-8763-b7bf9111d163" containerName="istio-proxy" Oct 31 03:18:07.423432 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.423390 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6980-caa51b" Oct 31 03:18:07.429177 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.429110 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bshq7\"" Oct 31 03:18:07.450165 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.450134 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6980-caa51b"] Oct 31 03:18:07.510493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.510449 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx755\" (UniqueName: \"kubernetes.io/projected/b1c31819-ba41-46ff-b91d-bc6c41084e32-kube-api-access-sx755\") pod \"test-curl-6980-caa51b\" (UID: \"b1c31819-ba41-46ff-b91d-bc6c41084e32\") " pod="opendatahub/test-curl-6980-caa51b" Oct 31 03:18:07.614814 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.614772 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx755\" (UniqueName: \"kubernetes.io/projected/b1c31819-ba41-46ff-b91d-bc6c41084e32-kube-api-access-sx755\") pod \"test-curl-6980-caa51b\" (UID: \"b1c31819-ba41-46ff-b91d-bc6c41084e32\") " pod="opendatahub/test-curl-6980-caa51b" Oct 31 03:18:07.695104 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.694617 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx755\" (UniqueName: \"kubernetes.io/projected/b1c31819-ba41-46ff-b91d-bc6c41084e32-kube-api-access-sx755\") pod \"test-curl-6980-caa51b\" (UID: \"b1c31819-ba41-46ff-b91d-bc6c41084e32\") " pod="opendatahub/test-curl-6980-caa51b" Oct 31 03:18:07.780089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:07.779563 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6980-caa51b" Oct 31 03:18:08.218981 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:18:08.218944 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1c31819_ba41_46ff_b91d_bc6c41084e32.slice/crio-507b3ac2aff592f002d47c5eb01dba3aa8958771e503d916c878d70fdd16d175 WatchSource:0}: Error finding container 507b3ac2aff592f002d47c5eb01dba3aa8958771e503d916c878d70fdd16d175: Status 404 returned error can't find the container with id 507b3ac2aff592f002d47c5eb01dba3aa8958771e503d916c878d70fdd16d175 Oct 31 03:18:08.222646 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:08.222619 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6980-caa51b"] Oct 31 03:18:08.874531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:08.874489 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6980-caa51b" event={"ID":"b1c31819-ba41-46ff-b91d-bc6c41084e32","Type":"ContainerStarted","Data":"3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52"} Oct 31 03:18:08.877554 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:08.877523 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6980-caa51b" event={"ID":"b1c31819-ba41-46ff-b91d-bc6c41084e32","Type":"ContainerStarted","Data":"507b3ac2aff592f002d47c5eb01dba3aa8958771e503d916c878d70fdd16d175"} Oct 31 03:18:08.901270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:08.901210 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6980-caa51b" podStartSLOduration=1.901178722 podStartE2EDuration="1.901178722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:08.900032485 +0000 UTC m=+1370.580558613" watchObservedRunningTime="2026-10-31 03:18:08.901178722 +0000 UTC m=+1370.581704850" Oct 31 03:18:09.905068 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:09.904135 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6980-caa51b"] Oct 31 03:18:09.910245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:09.910212 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6980-caa51b"] Oct 31 03:18:10.209624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.209528 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68"] Oct 31 03:18:10.210952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.210924 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b1c31819-ba41-46ff-b91d-bc6c41084e32" containerName="test-curl-6980-caa51b" Oct 31 03:18:10.210952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.210949 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c31819-ba41-46ff-b91d-bc6c41084e32" containerName="test-curl-6980-caa51b" Oct 31 03:18:10.213630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.211189 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c31819-ba41-46ff-b91d-bc6c41084e32" containerName="test-curl-6980-caa51b" Oct 31 03:18:10.215968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.215947 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:10.224762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.224729 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68"] Oct 31 03:18:10.225322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.224973 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:18:10.225595 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.225545 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:18:10.369418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.369379 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:10.369626 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.369609 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx8wx\" (UniqueName: \"kubernetes.io/projected/6445e06e-0035-4955-b6a7-f89d861997fa-kube-api-access-xx8wx\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:10.369727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.369684 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:10.480676 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.480584 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx8wx\" (UniqueName: \"kubernetes.io/projected/6445e06e-0035-4955-b6a7-f89d861997fa-kube-api-access-xx8wx\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:10.481339 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.481311 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:10.482236 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.482178 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:10.483333 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.483026 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:10.485453 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.485431 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:10.519057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.514863 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx8wx\" (UniqueName: \"kubernetes.io/projected/6445e06e-0035-4955-b6a7-f89d861997fa-kube-api-access-xx8wx\") pod \"maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:10.553111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:10.552374 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:11.058282 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:11.058228 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68"] Oct 31 03:18:11.200714 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:11.192697 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m"] Oct 31 03:18:11.200964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:11.200930 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" podUID="821d763a-861c-4d02-96b9-39aa69554d1a" containerName="maas-api" containerID="cri-o://0a496eb301e00234bc3e7fd51bdef0dd4c063fc34570f4ce375f5386b8c12b0c" gracePeriod=30 Oct 31 03:18:11.705958 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:11.705407 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" podUID="490d58df-8d11-4c30-9d01-55f5540c490e" containerName="payload-pre-processing" containerID="cri-o://e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30" gracePeriod=25 Oct 31 03:18:11.987727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:11.983180 4456 generic.go:356] "Generic (PLEG): container finished" podID="821d763a-861c-4d02-96b9-39aa69554d1a" containerID="0a496eb301e00234bc3e7fd51bdef0dd4c063fc34570f4ce375f5386b8c12b0c" exitCode=0 Oct 31 03:18:11.987727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:11.983445 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" event={"ID":"821d763a-861c-4d02-96b9-39aa69554d1a","Type":"ContainerDied","Data":"0a496eb301e00234bc3e7fd51bdef0dd4c063fc34570f4ce375f5386b8c12b0c"} Oct 31 03:18:11.987727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:11.983500 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" event={"ID":"821d763a-861c-4d02-96b9-39aa69554d1a","Type":"ContainerDied","Data":"c19f620bd5c60a0dbef44e7c67522c2140b930b0e4720469353443cabb43a04e"} Oct 31 03:18:11.987727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:11.983526 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c19f620bd5c60a0dbef44e7c67522c2140b930b0e4720469353443cabb43a04e" Oct 31 03:18:12.013368 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.010952 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6980-caa51b" podUID="b1c31819-ba41-46ff-b91d-bc6c41084e32" containerName="test-curl-6980-caa51b" containerID="cri-o://3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52" gracePeriod=2 Oct 31 03:18:12.013368 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.012202 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" event={"ID":"6445e06e-0035-4955-b6a7-f89d861997fa","Type":"ContainerStarted","Data":"95958d02f81007912ac85430f76060b619904501f32b4f1bc97153600efd5639"} Oct 31 03:18:12.016928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.016908 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:18:12.026203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.026167 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8"] Oct 31 03:18:12.032173 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.031393 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-56afbd-q24n8"] Oct 31 03:18:12.125185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.108426 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821d763a-861c-4d02-96b9-39aa69554d1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821d763a-861c-4d02-96b9-39aa69554d1a-openshift-service-ca\") pod \"821d763a-861c-4d02-96b9-39aa69554d1a\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " Oct 31 03:18:12.125185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.108661 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821d763a-861c-4d02-96b9-39aa69554d1a-kube-api-access-nmklp\" (UniqueName: \"kubernetes.io/projected/821d763a-861c-4d02-96b9-39aa69554d1a-kube-api-access-nmklp\") pod \"821d763a-861c-4d02-96b9-39aa69554d1a\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " Oct 31 03:18:12.125185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.108698 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" podStartSLOduration=2.108678712 podStartE2EDuration="2.108678712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:12.038553448 +0000 UTC m=+1373.719079575" watchObservedRunningTime="2026-10-31 03:18:12.108678712 +0000 UTC m=+1373.789204840" Oct 31 03:18:12.125185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.112684 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821d763a-861c-4d02-96b9-39aa69554d1a-kube-api-access-nmklp" pod "821d763a-861c-4d02-96b9-39aa69554d1a" (UID: "821d763a-861c-4d02-96b9-39aa69554d1a"). InnerVolumeSpecName "kube-api-access-nmklp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:12.128344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.126153 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821d763a-861c-4d02-96b9-39aa69554d1a-openshift-service-ca" pod "821d763a-861c-4d02-96b9-39aa69554d1a" (UID: "821d763a-861c-4d02-96b9-39aa69554d1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:12.215301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.212079 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-metrics-certs\") pod \"821d763a-861c-4d02-96b9-39aa69554d1a\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " Oct 31 03:18:12.215301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.212177 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-maas-api-tls\") pod \"821d763a-861c-4d02-96b9-39aa69554d1a\" (UID: \"821d763a-861c-4d02-96b9-39aa69554d1a\") " Oct 31 03:18:12.215301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.213001 4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821d763a-861c-4d02-96b9-39aa69554d1a-openshift-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:12.215301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.213024 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmklp\" (UniqueName: \"kubernetes.io/projected/821d763a-861c-4d02-96b9-39aa69554d1a-kube-api-access-nmklp\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:12.230595 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.230538 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-maas-api-tls" pod "821d763a-861c-4d02-96b9-39aa69554d1a" (UID: "821d763a-861c-4d02-96b9-39aa69554d1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:12.230813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.230657 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-metrics-certs" pod "821d763a-861c-4d02-96b9-39aa69554d1a" (UID: "821d763a-861c-4d02-96b9-39aa69554d1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:12.316129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.315511 4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-metrics-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:12.316129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.315547 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821d763a-861c-4d02-96b9-39aa69554d1a-maas-api-tls\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:12.422501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.422474 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:18:12.439104 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.439073 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b5c830-aacb-4b6f-9e1e-f4ac1393731a" path="/var/lib/kubelet/pods/33b5c830-aacb-4b6f-9e1e-f4ac1393731a/volumes" Oct 31 03:18:12.519378 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.519335 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490d58df-8d11-4c30-9d01-55f5540c490e-kube-api-access-4klvp\" (UniqueName: \"kubernetes.io/projected/490d58df-8d11-4c30-9d01-55f5540c490e-kube-api-access-4klvp\") pod \"490d58df-8d11-4c30-9d01-55f5540c490e\" (UID: \"490d58df-8d11-4c30-9d01-55f5540c490e\") " Oct 31 03:18:12.520249 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.520222 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490d58df-8d11-4c30-9d01-55f5540c490e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490d58df-8d11-4c30-9d01-55f5540c490e-plugins-config-volume\") pod \"490d58df-8d11-4c30-9d01-55f5540c490e\" (UID: \"490d58df-8d11-4c30-9d01-55f5540c490e\") " Oct 31 03:18:12.521479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.521457 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490d58df-8d11-4c30-9d01-55f5540c490e-plugins-config-volume" pod "490d58df-8d11-4c30-9d01-55f5540c490e" (UID: "490d58df-8d11-4c30-9d01-55f5540c490e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:12.522373 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.521717 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490d58df-8d11-4c30-9d01-55f5540c490e-kube-api-access-4klvp" pod "490d58df-8d11-4c30-9d01-55f5540c490e" (UID: "490d58df-8d11-4c30-9d01-55f5540c490e"). InnerVolumeSpecName "kube-api-access-4klvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:12.622750 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.622693 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490d58df-8d11-4c30-9d01-55f5540c490e-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:12.622750 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:12.622725 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4klvp\" (UniqueName: \"kubernetes.io/projected/490d58df-8d11-4c30-9d01-55f5540c490e-kube-api-access-4klvp\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:13.021837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.021801 4456 generic.go:356] "Generic (PLEG): container finished" podID="6445e06e-0035-4955-b6a7-f89d861997fa" containerID="ba23a9ed464c5afd7da3494849804e7936a5ed14cf73aa6d4cade2c703d2e806" exitCode=0 Oct 31 03:18:13.022008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.021884 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" event={"ID":"6445e06e-0035-4955-b6a7-f89d861997fa","Type":"ContainerDied","Data":"ba23a9ed464c5afd7da3494849804e7936a5ed14cf73aa6d4cade2c703d2e806"} Oct 31 03:18:13.023478 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.023456 4456 generic.go:356] "Generic (PLEG): container finished" podID="490d58df-8d11-4c30-9d01-55f5540c490e" containerID="e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30" exitCode=0 Oct 31 03:18:13.023587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.023501 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" Oct 31 03:18:13.023587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.023541 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" event={"ID":"490d58df-8d11-4c30-9d01-55f5540c490e","Type":"ContainerDied","Data":"e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30"} Oct 31 03:18:13.023681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.023588 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx" event={"ID":"490d58df-8d11-4c30-9d01-55f5540c490e","Type":"ContainerDied","Data":"872b88877b117bcd4d554d760c6d04476e3810f4c033f4cad49fd67bc23f855d"} Oct 31 03:18:13.023681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.023615 4456 scope.go:122] "RemoveContainer" containerID="e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30" Oct 31 03:18:13.023760 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.023707 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" Oct 31 03:18:13.036169 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.036149 4456 scope.go:122] "RemoveContainer" containerID="e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30" Oct 31 03:18:13.036437 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:18:13.036420 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30\": container with ID starting with e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30 not found: ID does not exist" containerID="e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30" Oct 31 03:18:13.036489 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.036444 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30"} err="failed to get container status \"e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30\": rpc error: code = NotFound desc = could not find container \"e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30\": container with ID starting with e4d86c001d3f065216233dd337ac0ed65f39af03524f3e5d6e538101d2f81a30 not found: ID does not exist" Oct 31 03:18:13.078191 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.069655 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m"] Oct 31 03:18:13.078191 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.077576 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m"] Oct 31 03:18:13.085310 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.085278 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx"] Oct 31 03:18:13.092746 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:13.092716 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-780abd-78c5ccff9-pm9rx"] Oct 31 03:18:14.243310 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.243266 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:14.284926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.274588 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-openshift-ca\") pod \"6445e06e-0035-4955-b6a7-f89d861997fa\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " Oct 31 03:18:14.284926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.274690 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-xks-ca\") pod \"6445e06e-0035-4955-b6a7-f89d861997fa\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " Oct 31 03:18:14.284926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.274865 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6445e06e-0035-4955-b6a7-f89d861997fa-kube-api-access-xx8wx\" (UniqueName: \"kubernetes.io/projected/6445e06e-0035-4955-b6a7-f89d861997fa-kube-api-access-xx8wx\") pod \"6445e06e-0035-4955-b6a7-f89d861997fa\" (UID: \"6445e06e-0035-4955-b6a7-f89d861997fa\") " Oct 31 03:18:14.284926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.281792 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-openshift-ca" pod "6445e06e-0035-4955-b6a7-f89d861997fa" (UID: "6445e06e-0035-4955-b6a7-f89d861997fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:14.284926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.284794 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-xks-ca" pod "6445e06e-0035-4955-b6a7-f89d861997fa" (UID: "6445e06e-0035-4955-b6a7-f89d861997fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:14.288884 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.288858 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6445e06e-0035-4955-b6a7-f89d861997fa-kube-api-access-xx8wx" pod "6445e06e-0035-4955-b6a7-f89d861997fa" (UID: "6445e06e-0035-4955-b6a7-f89d861997fa"). InnerVolumeSpecName "kube-api-access-xx8wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:14.300444 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.300409 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6980-caa51b" Oct 31 03:18:14.376382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.376350 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c31819-ba41-46ff-b91d-bc6c41084e32-kube-api-access-sx755\" (UniqueName: \"kubernetes.io/projected/b1c31819-ba41-46ff-b91d-bc6c41084e32-kube-api-access-sx755\") pod \"b1c31819-ba41-46ff-b91d-bc6c41084e32\" (UID: \"b1c31819-ba41-46ff-b91d-bc6c41084e32\") " Oct 31 03:18:14.377117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.377093 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.377117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.377118 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6445e06e-0035-4955-b6a7-f89d861997fa-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.377333 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.377134 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx8wx\" (UniqueName: \"kubernetes.io/projected/6445e06e-0035-4955-b6a7-f89d861997fa-kube-api-access-xx8wx\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.378765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.378720 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c31819-ba41-46ff-b91d-bc6c41084e32-kube-api-access-sx755" pod "b1c31819-ba41-46ff-b91d-bc6c41084e32" (UID: "b1c31819-ba41-46ff-b91d-bc6c41084e32"). InnerVolumeSpecName "kube-api-access-sx755". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:14.446435 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.446384 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490d58df-8d11-4c30-9d01-55f5540c490e" path="/var/lib/kubelet/pods/490d58df-8d11-4c30-9d01-55f5540c490e/volumes" Oct 31 03:18:14.447804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.447775 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821d763a-861c-4d02-96b9-39aa69554d1a" path="/var/lib/kubelet/pods/821d763a-861c-4d02-96b9-39aa69554d1a/volumes" Oct 31 03:18:14.448585 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.448556 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c31819-ba41-46ff-b91d-bc6c41084e32" path="/var/lib/kubelet/pods/b1c31819-ba41-46ff-b91d-bc6c41084e32/volumes" Oct 31 03:18:14.479761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.479730 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx755\" (UniqueName: \"kubernetes.io/projected/b1c31819-ba41-46ff-b91d-bc6c41084e32-kube-api-access-sx755\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.726455 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:14.726409 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56afbd-74d777bcf8-r9z7m" podUID="821d763a-861c-4d02-96b9-39aa69554d1a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.56:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:15.079399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:15.079360 4456 generic.go:356] "Generic (PLEG): container finished" podID="b1c31819-ba41-46ff-b91d-bc6c41084e32" containerID="3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52" exitCode=137 Oct 31 03:18:15.082426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:15.082403 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" Oct 31 03:18:15.090410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:15.084201 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6980-caa51b" Oct 31 03:18:15.090410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:15.084988 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68" event={"ID":"6445e06e-0035-4955-b6a7-f89d861997fa","Type":"ContainerDied","Data":"95958d02f81007912ac85430f76060b619904501f32b4f1bc97153600efd5639"} Oct 31 03:18:15.090410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:15.085054 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95958d02f81007912ac85430f76060b619904501f32b4f1bc97153600efd5639" Oct 31 03:18:15.090410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:15.085075 4456 scope.go:122] "RemoveContainer" containerID="3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52" Oct 31 03:18:15.133385 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:15.133333 4456 scope.go:122] "RemoveContainer" containerID="3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52" Oct 31 03:18:15.134101 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:18:15.134015 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52\": container with ID starting with 3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52 not found: ID does not exist" containerID="3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52" Oct 31 03:18:15.134219 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:15.134125 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52"} err="failed to get container status \"3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52\": rpc error: code = NotFound desc = could not find container \"3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52\": container with ID starting with 3a6bed766d70b2ad5246f1b7882df65e3433c392ea1e0516477bb9b946e59b52 not found: ID does not exist" Oct 31 03:18:18.953842 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:18.953803 4456 scope.go:122] "RemoveContainer" containerID="0a496eb301e00234bc3e7fd51bdef0dd4c063fc34570f4ce375f5386b8c12b0c" Oct 31 03:18:18.969310 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:18.969282 4456 scope.go:122] "RemoveContainer" containerID="c4c8421100e8ebf558c6bad35be960f5153b671974ec9f2813e763ae25611b0f" Oct 31 03:18:22.142764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:22.141096 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc"] Oct 31 03:18:22.142764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:22.141403 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" containerID="cri-o://1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110" gracePeriod=30 Oct 31 03:18:22.197059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:22.144760 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc"] Oct 31 03:18:22.197059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:22.179498 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" containerID="cri-o://08f03631154d59f0f7f994e61019e9139c04fb20c85b86bd50a5c717bfea7a85" gracePeriod=30 Oct 31 03:18:22.797061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:22.792161 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:23.296380 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:23.296302 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 31 03:18:23.619118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:23.608166 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:26.839760 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:26.839723 4456 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:18:26.840303 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:26.839781 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:18:27.500888 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.500007 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:18:27.522500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.522460 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-envoy\") pod \"aa8ce63a-cace-42b0-891c-836915b09ca7\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " Oct 31 03:18:27.522664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.522632 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-credential-socket\") pod \"aa8ce63a-cace-42b0-891c-836915b09ca7\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " Oct 31 03:18:27.522740 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.522725 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-data\") pod \"aa8ce63a-cace-42b0-891c-836915b09ca7\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " Oct 31 03:18:27.522782 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.522769 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-socket\") pod \"aa8ce63a-cace-42b0-891c-836915b09ca7\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " Oct 31 03:18:27.522923 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.522900 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-certs\") pod \"aa8ce63a-cace-42b0-891c-836915b09ca7\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " Oct 31 03:18:27.523150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.522982 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-kube-api-access-7gx2z\" (UniqueName: \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-kube-api-access-7gx2z\") pod \"aa8ce63a-cace-42b0-891c-836915b09ca7\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " Oct 31 03:18:27.523150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.523010 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-podinfo\") pod \"aa8ce63a-cace-42b0-891c-836915b09ca7\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " Oct 31 03:18:27.523150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.523098 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8ce63a-cace-42b0-891c-836915b09ca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8ce63a-cace-42b0-891c-836915b09ca7-istiod-ca-cert\") pod \"aa8ce63a-cace-42b0-891c-836915b09ca7\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " Oct 31 03:18:27.523150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.523144 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-token\" (UniqueName: \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-token\") pod \"aa8ce63a-cace-42b0-891c-836915b09ca7\" (UID: \"aa8ce63a-cace-42b0-891c-836915b09ca7\") " Oct 31 03:18:27.523397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.523139 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-credential-socket" pod "aa8ce63a-cace-42b0-891c-836915b09ca7" (UID: "aa8ce63a-cace-42b0-891c-836915b09ca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.523976 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.523945 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.524554 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.524526 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-data" pod "aa8ce63a-cace-42b0-891c-836915b09ca7" (UID: "aa8ce63a-cace-42b0-891c-836915b09ca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.524946 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.524895 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-certs" pod "aa8ce63a-cace-42b0-891c-836915b09ca7" (UID: "aa8ce63a-cace-42b0-891c-836915b09ca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.525094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.525067 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-envoy" pod "aa8ce63a-cace-42b0-891c-836915b09ca7" (UID: "aa8ce63a-cace-42b0-891c-836915b09ca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.525450 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.525124 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-socket" pod "aa8ce63a-cace-42b0-891c-836915b09ca7" (UID: "aa8ce63a-cace-42b0-891c-836915b09ca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.525450 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.525308 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8ce63a-cace-42b0-891c-836915b09ca7-istiod-ca-cert" pod "aa8ce63a-cace-42b0-891c-836915b09ca7" (UID: "aa8ce63a-cace-42b0-891c-836915b09ca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:27.526644 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.526623 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-podinfo" pod "aa8ce63a-cace-42b0-891c-836915b09ca7" (UID: "aa8ce63a-cace-42b0-891c-836915b09ca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:27.527695 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.527668 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-kube-api-access-7gx2z" pod "aa8ce63a-cace-42b0-891c-836915b09ca7" (UID: "aa8ce63a-cace-42b0-891c-836915b09ca7"). InnerVolumeSpecName "kube-api-access-7gx2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:27.529058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.528207 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-token" pod "aa8ce63a-cace-42b0-891c-836915b09ca7" (UID: "aa8ce63a-cace-42b0-891c-836915b09ca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:27.583486 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.583452 4456 generic.go:356] "Generic (PLEG): container finished" podID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerID="08f03631154d59f0f7f994e61019e9139c04fb20c85b86bd50a5c717bfea7a85" exitCode=0 Oct 31 03:18:27.583638 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.583531 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" event={"ID":"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3","Type":"ContainerDied","Data":"08f03631154d59f0f7f994e61019e9139c04fb20c85b86bd50a5c717bfea7a85"} Oct 31 03:18:27.590159 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.590123 4456 generic.go:356] "Generic (PLEG): container finished" podID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerID="1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110" exitCode=0 Oct 31 03:18:27.590455 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.590432 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" event={"ID":"aa8ce63a-cace-42b0-891c-836915b09ca7","Type":"ContainerDied","Data":"1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110"} Oct 31 03:18:27.590533 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.590491 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" event={"ID":"aa8ce63a-cace-42b0-891c-836915b09ca7","Type":"ContainerDied","Data":"9cf77f54aff1c38049092d6067f93dcb905557d48376182f904e3d9384303048"} Oct 31 03:18:27.590533 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.590526 4456 scope.go:122] "RemoveContainer" containerID="1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110" Oct 31 03:18:27.590704 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.590686 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc" Oct 31 03:18:27.602004 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.601974 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:18:27.611230 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.611209 4456 scope.go:122] "RemoveContainer" containerID="1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110" Oct 31 03:18:27.611524 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:18:27.611501 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110\": container with ID starting with 1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110 not found: ID does not exist" containerID="1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110" Oct 31 03:18:27.611587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.611533 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110"} err="failed to get container status \"1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110\": rpc error: code = NotFound desc = could not find container \"1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110\": container with ID starting with 1f75e98fd28809838af084b720e1039576517337d31f4db454b7955741b84110 not found: ID does not exist" Oct 31 03:18:27.625368 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.625335 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-envoy\") pod \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " Oct 31 03:18:27.625517 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.625406 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-certs\") pod \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " Oct 31 03:18:27.625517 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.625442 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-token\" (UniqueName: \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-token\") pod \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " Oct 31 03:18:27.625637 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.625527 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istiod-ca-cert\") pod \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " Oct 31 03:18:27.625704 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.625686 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-certs" pod "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" (UID: "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.625883 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.625864 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istiod-ca-cert" pod "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" (UID: "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:27.625947 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.625923 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-data\") pod \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " Oct 31 03:18:27.626020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.625991 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-kube-api-access-pm9lb\" (UniqueName: \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-kube-api-access-pm9lb\") pod \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " Oct 31 03:18:27.626103 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.626083 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-socket\") pod \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " Oct 31 03:18:27.626232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.626206 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-data" pod "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" (UID: "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.626386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.626367 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-socket" pod "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" (UID: "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.626469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.626453 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-credential-socket\") pod \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " Oct 31 03:18:27.626534 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.626490 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-podinfo\") pod \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\" (UID: \"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3\") " Oct 31 03:18:27.627032 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.626698 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-credential-socket" pod "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" (UID: "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.627193 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627174 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627263 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627198 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627263 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627230 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627263 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627246 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gx2z\" (UniqueName: \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-kube-api-access-7gx2z\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627263 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627280 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8ce63a-cace-42b0-891c-836915b09ca7-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627294 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627308 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627321 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627333 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627346 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627365 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ce63a-cace-42b0-891c-836915b09ca7-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.627448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627379 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.630100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.627734 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-envoy" pod "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" (UID: "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.630100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.629063 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-podinfo" pod "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" (UID: "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:27.630100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.629364 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-token" pod "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" (UID: "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:27.630294 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.630171 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-kube-api-access-pm9lb" pod "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" (UID: "c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3"). InnerVolumeSpecName "kube-api-access-pm9lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:27.648832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.648801 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc"] Oct 31 03:18:27.665203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.665172 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tpmgc"] Oct 31 03:18:27.729138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.729101 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.729138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.729133 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm9lb\" (UniqueName: \"kubernetes.io/projected/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-kube-api-access-pm9lb\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.729138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.729144 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.729369 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:27.729153 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.438075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:28.438021 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" path="/var/lib/kubelet/pods/aa8ce63a-cace-42b0-891c-836915b09ca7/volumes" Oct 31 03:18:28.632461 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:28.632366 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" Oct 31 03:18:28.633194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:28.633161 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc" event={"ID":"c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3","Type":"ContainerDied","Data":"18d1ee6d8e277e45c718085db9de4f7d863cd11482fdae7dd701ae560c13cc73"} Oct 31 03:18:28.633370 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:28.633236 4456 scope.go:122] "RemoveContainer" containerID="08f03631154d59f0f7f994e61019e9139c04fb20c85b86bd50a5c717bfea7a85" Oct 31 03:18:28.663445 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:28.663326 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc"] Oct 31 03:18:28.668382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:28.668351 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-fp4wc"] Oct 31 03:18:30.439154 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:30.439118 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" path="/var/lib/kubelet/pods/c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3/volumes" Oct 31 03:18:31.881646 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.881610 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m"] Oct 31 03:18:31.882586 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882564 4456 cpu_manager.go:414] "containerMap: removing container" podUID="490d58df-8d11-4c30-9d01-55f5540c490e" containerName="payload-pre-processing" Oct 31 03:18:31.882586 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882586 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490d58df-8d11-4c30-9d01-55f5540c490e" containerName="payload-pre-processing" Oct 31 03:18:31.882768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882624 4456 cpu_manager.go:414] "containerMap: removing container" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" Oct 31 03:18:31.882768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882633 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" Oct 31 03:18:31.882768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882646 4456 cpu_manager.go:414] "containerMap: removing container" podUID="6445e06e-0035-4955-b6a7-f89d861997fa" containerName="revoke-keys" Oct 31 03:18:31.882768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882653 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6445e06e-0035-4955-b6a7-f89d861997fa" containerName="revoke-keys" Oct 31 03:18:31.882768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882681 4456 cpu_manager.go:414] "containerMap: removing container" podUID="821d763a-861c-4d02-96b9-39aa69554d1a" containerName="maas-api" Oct 31 03:18:31.882768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882688 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821d763a-861c-4d02-96b9-39aa69554d1a" containerName="maas-api" Oct 31 03:18:31.882768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882712 4456 cpu_manager.go:414] "containerMap: removing container" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" Oct 31 03:18:31.882768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882720 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" Oct 31 03:18:31.883181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882903 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="821d763a-861c-4d02-96b9-39aa69554d1a" containerName="maas-api" Oct 31 03:18:31.883181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882920 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8ce63a-cace-42b0-891c-836915b09ca7" containerName="istio-proxy" Oct 31 03:18:31.883181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882938 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="490d58df-8d11-4c30-9d01-55f5540c490e" containerName="payload-pre-processing" Oct 31 03:18:31.883181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882954 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e5f3ec-ce41-47bd-8d75-bce776c5a8c3" containerName="istio-proxy" Oct 31 03:18:31.883181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.882967 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6445e06e-0035-4955-b6a7-f89d861997fa" containerName="revoke-keys" Oct 31 03:18:31.888962 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.888941 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:31.893465 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.893438 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m"] Oct 31 03:18:31.893901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.893886 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:18:31.895832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:31.895811 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:18:32.034469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.034424 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:32.034677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.034559 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:32.034807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.034790 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2pfs\" (UniqueName: \"kubernetes.io/projected/8f3dc910-ee71-4521-bd98-3461b549c935-kube-api-access-z2pfs\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:32.136567 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.136486 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pfs\" (UniqueName: \"kubernetes.io/projected/8f3dc910-ee71-4521-bd98-3461b549c935-kube-api-access-z2pfs\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:32.136567 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.136553 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:32.136741 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.136618 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:32.137376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.137345 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:32.137669 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:18:32.137616 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:18:32.139282 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.139256 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:32.149375 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.149346 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pfs\" (UniqueName: \"kubernetes.io/projected/8f3dc910-ee71-4521-bd98-3461b549c935-kube-api-access-z2pfs\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:32.212684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.212653 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:32.504735 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:18:32.504688 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3dc910_ee71_4521_bd98_3461b549c935.slice/crio-147035ae04d0067975438b7d52a3080d5d01bbf1843ef8870d3ef25f7f443f09 WatchSource:0}: Error finding container 147035ae04d0067975438b7d52a3080d5d01bbf1843ef8870d3ef25f7f443f09: Status 404 returned error can't find the container with id 147035ae04d0067975438b7d52a3080d5d01bbf1843ef8870d3ef25f7f443f09 Oct 31 03:18:32.505593 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.505476 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m"] Oct 31 03:18:32.741234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:32.741198 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" event={"ID":"8f3dc910-ee71-4521-bd98-3461b549c935","Type":"ContainerStarted","Data":"147035ae04d0067975438b7d52a3080d5d01bbf1843ef8870d3ef25f7f443f09"} Oct 31 03:18:33.795617 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:33.795521 4456 generic.go:356] "Generic (PLEG): container finished" podID="8f3dc910-ee71-4521-bd98-3461b549c935" containerID="12d9a24bf845dc5d32d994d10a7afcbbdc4e1e7d8df96796102ca018334bf39d" exitCode=0 Oct 31 03:18:33.797174 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:33.796912 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" event={"ID":"8f3dc910-ee71-4521-bd98-3461b549c935","Type":"ContainerDied","Data":"12d9a24bf845dc5d32d994d10a7afcbbdc4e1e7d8df96796102ca018334bf39d"} Oct 31 03:18:35.038830 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.038802 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:35.116204 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.116108 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-xks-ca\") pod \"8f3dc910-ee71-4521-bd98-3461b549c935\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " Oct 31 03:18:35.116382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.116208 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-openshift-ca\") pod \"8f3dc910-ee71-4521-bd98-3461b549c935\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " Oct 31 03:18:35.116382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.116352 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3dc910-ee71-4521-bd98-3461b549c935-kube-api-access-z2pfs\" (UniqueName: \"kubernetes.io/projected/8f3dc910-ee71-4521-bd98-3461b549c935-kube-api-access-z2pfs\") pod \"8f3dc910-ee71-4521-bd98-3461b549c935\" (UID: \"8f3dc910-ee71-4521-bd98-3461b549c935\") " Oct 31 03:18:35.116885 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.116851 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-openshift-ca" pod "8f3dc910-ee71-4521-bd98-3461b549c935" (UID: "8f3dc910-ee71-4521-bd98-3461b549c935"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:35.118847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.118820 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-xks-ca" pod "8f3dc910-ee71-4521-bd98-3461b549c935" (UID: "8f3dc910-ee71-4521-bd98-3461b549c935"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:35.118993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.118975 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3dc910-ee71-4521-bd98-3461b549c935-kube-api-access-z2pfs" pod "8f3dc910-ee71-4521-bd98-3461b549c935" (UID: "8f3dc910-ee71-4521-bd98-3461b549c935"). InnerVolumeSpecName "kube-api-access-z2pfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:35.218630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.218594 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2pfs\" (UniqueName: \"kubernetes.io/projected/8f3dc910-ee71-4521-bd98-3461b549c935-kube-api-access-z2pfs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:35.218630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.218625 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:35.218630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.218636 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f3dc910-ee71-4521-bd98-3461b549c935-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:35.838317 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.838276 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" event={"ID":"8f3dc910-ee71-4521-bd98-3461b549c935","Type":"ContainerDied","Data":"147035ae04d0067975438b7d52a3080d5d01bbf1843ef8870d3ef25f7f443f09"} Oct 31 03:18:35.838484 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.838343 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="147035ae04d0067975438b7d52a3080d5d01bbf1843ef8870d3ef25f7f443f09" Oct 31 03:18:35.838484 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:35.838413 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m" Oct 31 03:18:36.694131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:36.694094 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68"] Oct 31 03:18:36.723795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:36.722022 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1bfbe55-rpv68"] Oct 31 03:18:38.444531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:38.444494 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6445e06e-0035-4955-b6a7-f89d861997fa" path="/var/lib/kubelet/pods/6445e06e-0035-4955-b6a7-f89d861997fa/volumes" Oct 31 03:18:42.113820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:42.113784 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt"] Oct 31 03:18:43.964757 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:43.964722 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv"] Oct 31 03:18:43.967394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:43.965022 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" containerID="cri-o://ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c" gracePeriod=30 Oct 31 03:18:49.260785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.260755 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:18:49.347813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.347112 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-kube-api-access-7qlsr\" (UniqueName: \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-kube-api-access-7qlsr\") pod \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " Oct 31 03:18:49.347813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.347315 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-podinfo\") pod \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " Oct 31 03:18:49.347813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.347379 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-envoy\") pod \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " Oct 31 03:18:49.347813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.347409 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-token\") pod \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " Oct 31 03:18:49.348196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.347872 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-data\") pod \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " Oct 31 03:18:49.348196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.347948 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istiod-ca-cert\") pod \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " Oct 31 03:18:49.348196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.347984 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-socket\") pod \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " Oct 31 03:18:49.348196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.348088 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-certs\") pod \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " Oct 31 03:18:49.348196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.348187 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-credential-socket\") pod \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\" (UID: \"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485\") " Oct 31 03:18:49.349241 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.349209 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-credential-socket" pod "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" (UID: "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:49.349450 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.349427 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-data" pod "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" (UID: "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:49.349828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.349805 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istiod-ca-cert" pod "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" (UID: "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:49.350094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.350071 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-socket" pod "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" (UID: "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:49.350293 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.350273 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-certs" pod "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" (UID: "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:49.351342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.351301 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-podinfo" pod "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" (UID: "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:49.352361 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.352270 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-token" pod "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" (UID: "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:49.353878 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.353844 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-kube-api-access-7qlsr" pod "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" (UID: "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485"). InnerVolumeSpecName "kube-api-access-7qlsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:49.358188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.358160 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-envoy" pod "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" (UID: "f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:49.450882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.450846 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qlsr\" (UniqueName: \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-kube-api-access-7qlsr\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:49.450882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.450880 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:49.450882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.450889 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:49.451164 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.450898 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:49.451164 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.450907 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:49.451164 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.450915 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:49.451164 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.450924 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:49.451164 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.450932 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:49.451164 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:49.450941 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.272242 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:50.272200 4456 generic.go:356] "Generic (PLEG): container finished" podID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerID="ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c" exitCode=0 Oct 31 03:18:50.272980 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:50.272836 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" Oct 31 03:18:50.276713 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:50.272856 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" event={"ID":"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485","Type":"ContainerDied","Data":"ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c"} Oct 31 03:18:50.276924 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:50.276900 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv" event={"ID":"f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485","Type":"ContainerDied","Data":"c94498c3639f71742aed0b1fec7f38529dd9d6223b3d495d94575c3d5e14d186"} Oct 31 03:18:50.277064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:50.277048 4456 scope.go:122] "RemoveContainer" containerID="ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c" Oct 31 03:18:50.362778 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:50.362716 4456 scope.go:122] "RemoveContainer" containerID="ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c" Oct 31 03:18:50.369320 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:18:50.369211 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c\": container with ID starting with ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c not found: ID does not exist" containerID="ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c" Oct 31 03:18:50.369320 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:50.369260 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c"} err="failed to get container status \"ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c\": rpc error: code = NotFound desc = could not find container \"ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c\": container with ID starting with ec8e4c3138bc575e28c75fae86aef2eae01c27ed06c702572c944d150125a02c not found: ID does not exist" Oct 31 03:18:50.393569 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:50.393537 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv"] Oct 31 03:18:50.423614 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:50.423559 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-xdbsv"] Oct 31 03:18:50.444906 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:50.444868 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" path="/var/lib/kubelet/pods/f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485/volumes" Oct 31 03:18:51.714271 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:51.714223 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m"] Oct 31 03:18:51.966296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:51.966197 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b93a15-b9z4m"] Oct 31 03:18:52.437820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:52.437785 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f3dc910-ee71-4521-bd98-3461b549c935" path="/var/lib/kubelet/pods/8f3dc910-ee71-4521-bd98-3461b549c935/volumes" Oct 31 03:18:57.146273 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.146223 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" containerID="cri-o://0d25bc2e76cc6e455f8d0e273d54de693fba7185155052aad19905397480f38c" gracePeriod=45 Oct 31 03:18:57.293659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.293615 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": dial tcp 10.133.0.106:8000: connect: connection refused" Oct 31 03:18:57.416607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.416577 4456 generic.go:356] "Generic (PLEG): container finished" podID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerID="0d25bc2e76cc6e455f8d0e273d54de693fba7185155052aad19905397480f38c" exitCode=0 Oct 31 03:18:57.416730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.416649 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" event={"ID":"5f227327-853e-4e94-ad44-ec954ff5ab4b","Type":"ContainerDied","Data":"0d25bc2e76cc6e455f8d0e273d54de693fba7185155052aad19905397480f38c"} Oct 31 03:18:57.416730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.416692 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" event={"ID":"5f227327-853e-4e94-ad44-ec954ff5ab4b","Type":"ContainerDied","Data":"f2d98f39f078f9968fd336ac855e18cdb05d41a604c4a35befecdbb62829a390"} Oct 31 03:18:57.416730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.416703 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2d98f39f078f9968fd336ac855e18cdb05d41a604c4a35befecdbb62829a390" Oct 31 03:18:57.438805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.438779 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:18:57.454315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.453490 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f227327-853e-4e94-ad44-ec954ff5ab4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f227327-853e-4e94-ad44-ec954ff5ab4b-tls-certs\") pod \"5f227327-853e-4e94-ad44-ec954ff5ab4b\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " Oct 31 03:18:57.454315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.453627 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-tmp-dir\") pod \"5f227327-853e-4e94-ad44-ec954ff5ab4b\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " Oct 31 03:18:57.454315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.453681 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-dshm\") pod \"5f227327-853e-4e94-ad44-ec954ff5ab4b\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " Oct 31 03:18:57.454315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.453733 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-model-cache\") pod \"5f227327-853e-4e94-ad44-ec954ff5ab4b\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " Oct 31 03:18:57.454315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.453826 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f227327-853e-4e94-ad44-ec954ff5ab4b-kube-api-access-sf77w\" (UniqueName: \"kubernetes.io/projected/5f227327-853e-4e94-ad44-ec954ff5ab4b-kube-api-access-sf77w\") pod \"5f227327-853e-4e94-ad44-ec954ff5ab4b\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " Oct 31 03:18:57.454315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.453886 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-home\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-home\") pod \"5f227327-853e-4e94-ad44-ec954ff5ab4b\" (UID: \"5f227327-853e-4e94-ad44-ec954ff5ab4b\") " Oct 31 03:18:57.454315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.453936 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-tmp-dir" pod "5f227327-853e-4e94-ad44-ec954ff5ab4b" (UID: "5f227327-853e-4e94-ad44-ec954ff5ab4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:57.454315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.454278 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-home" pod "5f227327-853e-4e94-ad44-ec954ff5ab4b" (UID: "5f227327-853e-4e94-ad44-ec954ff5ab4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:57.454842 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.454459 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-model-cache" pod "5f227327-853e-4e94-ad44-ec954ff5ab4b" (UID: "5f227327-853e-4e94-ad44-ec954ff5ab4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:57.454842 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.454764 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:57.454842 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.454784 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:57.454842 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.454797 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:57.458556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.458524 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f227327-853e-4e94-ad44-ec954ff5ab4b-kube-api-access-sf77w" pod "5f227327-853e-4e94-ad44-ec954ff5ab4b" (UID: "5f227327-853e-4e94-ad44-ec954ff5ab4b"). InnerVolumeSpecName "kube-api-access-sf77w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:57.458671 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.458578 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-dshm" pod "5f227327-853e-4e94-ad44-ec954ff5ab4b" (UID: "5f227327-853e-4e94-ad44-ec954ff5ab4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:57.464660 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.464371 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f227327-853e-4e94-ad44-ec954ff5ab4b-tls-certs" pod "5f227327-853e-4e94-ad44-ec954ff5ab4b" (UID: "5f227327-853e-4e94-ad44-ec954ff5ab4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:57.556605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.556569 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f227327-853e-4e94-ad44-ec954ff5ab4b-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:57.556605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.556600 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f227327-853e-4e94-ad44-ec954ff5ab4b-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:57.556605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:57.556610 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf77w\" (UniqueName: \"kubernetes.io/projected/5f227327-853e-4e94-ad44-ec954ff5ab4b-kube-api-access-sf77w\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:58.425720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:58.425678 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt" Oct 31 03:18:58.456562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:58.456519 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt"] Oct 31 03:18:58.461230 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:18:58.461199 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f1bfbe55/e2e-lifecycle-model-f1bfbe55-kserve-7f7d768785-9w5vt"] Oct 31 03:19:00.469365 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:00.469333 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" path="/var/lib/kubelet/pods/5f227327-853e-4e94-ad44-ec954ff5ab4b/volumes" Oct 31 03:19:01.315572 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.315530 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v"] Oct 31 03:19:01.316738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.316714 4456 cpu_manager.go:414] "containerMap: removing container" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" Oct 31 03:19:01.316855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.316741 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" Oct 31 03:19:01.316855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.316775 4456 cpu_manager.go:414] "containerMap: removing container" podUID="8f3dc910-ee71-4521-bd98-3461b549c935" containerName="revoke-keys" Oct 31 03:19:01.316855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.316784 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f3dc910-ee71-4521-bd98-3461b549c935" containerName="revoke-keys" Oct 31 03:19:01.316855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.316813 4456 cpu_manager.go:414] "containerMap: removing container" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" Oct 31 03:19:01.316855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.316821 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" Oct 31 03:19:01.317015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.316997 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f227327-853e-4e94-ad44-ec954ff5ab4b" containerName="main" Oct 31 03:19:01.317062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.317014 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f3dc910-ee71-4521-bd98-3461b549c935" containerName="revoke-keys" Oct 31 03:19:01.317062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.317026 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f9a80b-b1f2-46e7-a9c8-2cbde2b47485" containerName="istio-proxy" Oct 31 03:19:01.322784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.322756 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:01.330405 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.330374 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v"] Oct 31 03:19:01.337159 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.337122 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:19:01.337441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.337416 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:19:01.536100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.527328 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:01.536100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.527466 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:01.536100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.527558 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9ddh\" (UniqueName: \"kubernetes.io/projected/aa71e91b-d850-4c83-82dd-5017cbe6c231-kube-api-access-g9ddh\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:01.629492 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.629384 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9ddh\" (UniqueName: \"kubernetes.io/projected/aa71e91b-d850-4c83-82dd-5017cbe6c231-kube-api-access-g9ddh\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:01.629966 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.629938 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:01.630159 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.630134 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:01.632276 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.632245 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:01.634503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.634473 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:01.644262 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.644236 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9ddh\" (UniqueName: \"kubernetes.io/projected/aa71e91b-d850-4c83-82dd-5017cbe6c231-kube-api-access-g9ddh\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:01.692910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:01.692865 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:02.062723 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:02.058335 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v"] Oct 31 03:19:02.168484 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:19:02.168449 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:19:02.552743 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:02.552707 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" event={"ID":"aa71e91b-d850-4c83-82dd-5017cbe6c231","Type":"ContainerStarted","Data":"b45826e827d99fb72d0a36f58a2e8920f6f40a6c0a185f96c7d71b1b927bbaf1"} Oct 31 03:19:03.563314 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:03.563215 4456 generic.go:356] "Generic (PLEG): container finished" podID="aa71e91b-d850-4c83-82dd-5017cbe6c231" containerID="dd8a13ceeee62949dd6b95c3b1c3a6e44e21ba6a3d6e4e88ad7468db0f72b341" exitCode=0 Oct 31 03:19:03.563314 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:03.563297 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" event={"ID":"aa71e91b-d850-4c83-82dd-5017cbe6c231","Type":"ContainerDied","Data":"dd8a13ceeee62949dd6b95c3b1c3a6e44e21ba6a3d6e4e88ad7468db0f72b341"} Oct 31 03:19:04.751624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:04.751592 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:04.796305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:04.796255 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-xks-ca\") pod \"aa71e91b-d850-4c83-82dd-5017cbe6c231\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " Oct 31 03:19:04.796487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:04.796404 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa71e91b-d850-4c83-82dd-5017cbe6c231-kube-api-access-g9ddh\" (UniqueName: \"kubernetes.io/projected/aa71e91b-d850-4c83-82dd-5017cbe6c231-kube-api-access-g9ddh\") pod \"aa71e91b-d850-4c83-82dd-5017cbe6c231\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " Oct 31 03:19:04.796487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:04.796432 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-openshift-ca\") pod \"aa71e91b-d850-4c83-82dd-5017cbe6c231\" (UID: \"aa71e91b-d850-4c83-82dd-5017cbe6c231\") " Oct 31 03:19:04.797131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:04.797099 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-openshift-ca" pod "aa71e91b-d850-4c83-82dd-5017cbe6c231" (UID: "aa71e91b-d850-4c83-82dd-5017cbe6c231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:04.797263 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:04.797247 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:04.798853 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:04.798828 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa71e91b-d850-4c83-82dd-5017cbe6c231-kube-api-access-g9ddh" pod "aa71e91b-d850-4c83-82dd-5017cbe6c231" (UID: "aa71e91b-d850-4c83-82dd-5017cbe6c231"). InnerVolumeSpecName "kube-api-access-g9ddh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:04.799275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:04.799251 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-xks-ca" pod "aa71e91b-d850-4c83-82dd-5017cbe6c231" (UID: "aa71e91b-d850-4c83-82dd-5017cbe6c231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:04.898695 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:04.898602 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9ddh\" (UniqueName: \"kubernetes.io/projected/aa71e91b-d850-4c83-82dd-5017cbe6c231-kube-api-access-g9ddh\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:04.898695 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:04.898636 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa71e91b-d850-4c83-82dd-5017cbe6c231-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:05.583446 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:05.583414 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" Oct 31 03:19:05.583653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:05.583443 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v" event={"ID":"aa71e91b-d850-4c83-82dd-5017cbe6c231","Type":"ContainerDied","Data":"b45826e827d99fb72d0a36f58a2e8920f6f40a6c0a185f96c7d71b1b927bbaf1"} Oct 31 03:19:05.583653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:05.583503 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b45826e827d99fb72d0a36f58a2e8920f6f40a6c0a185f96c7d71b1b927bbaf1" Oct 31 03:19:12.679756 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:12.679718 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v"] Oct 31 03:19:12.686711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:12.686677 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d1a8e-dbd9v"] Oct 31 03:19:14.378346 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:14.378307 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb"] Oct 31 03:19:14.378804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:14.378555 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" containerID="cri-o://d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525" gracePeriod=30 Oct 31 03:19:14.695063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:14.690953 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa71e91b-d850-4c83-82dd-5017cbe6c231" path="/var/lib/kubelet/pods/aa71e91b-d850-4c83-82dd-5017cbe6c231/volumes" Oct 31 03:19:14.731513 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:14.731480 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5"] Oct 31 03:19:14.731738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:14.731700 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" containerID="cri-o://428aabd3bda49291197b74e2dc70a3ecd615dfcfc45f3734d342995c02464a38" gracePeriod=30 Oct 31 03:19:17.342160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.342125 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv"] Oct 31 03:19:17.342918 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.342902 4456 cpu_manager.go:414] "containerMap: removing container" podUID="aa71e91b-d850-4c83-82dd-5017cbe6c231" containerName="revoke-keys" Oct 31 03:19:17.342975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.342919 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa71e91b-d850-4c83-82dd-5017cbe6c231" containerName="revoke-keys" Oct 31 03:19:17.343029 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.343019 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa71e91b-d850-4c83-82dd-5017cbe6c231" containerName="revoke-keys" Oct 31 03:19:17.346958 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.346936 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:19:17.352552 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.352526 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7269ae\"" Oct 31 03:19:17.359032 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.359009 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7269ae-dockercfg-mdhqf\"" Oct 31 03:19:17.366468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.366441 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv"] Oct 31 03:19:17.400964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.400932 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnhbw\" (UniqueName: \"kubernetes.io/projected/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-kube-api-access-wnhbw\") pod \"payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv\" (UID: \"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:19:17.401185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.400982 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv\" (UID: \"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:19:17.504188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.504149 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv\" (UID: \"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:19:17.549012 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.505318 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv\" (UID: \"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:19:17.553583 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.553549 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnhbw\" (UniqueName: \"kubernetes.io/projected/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-kube-api-access-wnhbw\") pod \"payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv\" (UID: \"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:19:17.606155 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.605806 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnhbw\" (UniqueName: \"kubernetes.io/projected/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-kube-api-access-wnhbw\") pod \"payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv\" (UID: \"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:19:17.682513 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:17.675314 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:19:18.061856 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:18.061828 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv"] Oct 31 03:19:18.911562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:18.911514 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" event={"ID":"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb","Type":"ContainerStarted","Data":"d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d"} Oct 31 03:19:18.912077 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:18.911596 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:19:18.912077 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:18.911623 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" event={"ID":"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb","Type":"ContainerStarted","Data":"909112f83202006b38080b56f504b0c8545b2c13540cbab271daa492be1d3f8c"} Oct 31 03:19:18.942699 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:18.942645 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" podStartSLOduration=1.942629138 podStartE2EDuration="1.942629138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:18.940197891 +0000 UTC m=+1440.620724018" watchObservedRunningTime="2026-10-31 03:19:18.942629138 +0000 UTC m=+1440.623155264" Oct 31 03:19:19.841070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.841023 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:19:19.902507 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.902458 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-podinfo\") pod \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " Oct 31 03:19:19.902707 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.902534 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istiod-ca-cert\") pod \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " Oct 31 03:19:19.902707 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.902585 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-kube-api-access-nhxwx\" (UniqueName: \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-kube-api-access-nhxwx\") pod \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " Oct 31 03:19:19.902707 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.902651 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-certs\") pod \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " Oct 31 03:19:19.902901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.902710 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-credential-socket\") pod \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " Oct 31 03:19:19.902901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.902754 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-envoy\") pod \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " Oct 31 03:19:19.902901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.902833 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-token\" (UniqueName: \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-token\") pod \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " Oct 31 03:19:19.903265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.903240 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-data\") pod \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " Oct 31 03:19:19.903855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.903822 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-credential-socket" pod "b99b5e71-0870-42ed-bd1a-302562ecaa8c" (UID: "b99b5e71-0870-42ed-bd1a-302562ecaa8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:19.904155 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.904127 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-certs" pod "b99b5e71-0870-42ed-bd1a-302562ecaa8c" (UID: "b99b5e71-0870-42ed-bd1a-302562ecaa8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:19.904489 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.904467 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-data" pod "b99b5e71-0870-42ed-bd1a-302562ecaa8c" (UID: "b99b5e71-0870-42ed-bd1a-302562ecaa8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:19.905098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.905060 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istiod-ca-cert" pod "b99b5e71-0870-42ed-bd1a-302562ecaa8c" (UID: "b99b5e71-0870-42ed-bd1a-302562ecaa8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:19.905821 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.905792 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:19.905928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.905825 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:19.905928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.905840 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:19.905928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.905855 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:19.908321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.908280 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-token" pod "b99b5e71-0870-42ed-bd1a-302562ecaa8c" (UID: "b99b5e71-0870-42ed-bd1a-302562ecaa8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:19.911102 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.911076 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-podinfo" pod "b99b5e71-0870-42ed-bd1a-302562ecaa8c" (UID: "b99b5e71-0870-42ed-bd1a-302562ecaa8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:19.912526 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.912497 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-envoy" pod "b99b5e71-0870-42ed-bd1a-302562ecaa8c" (UID: "b99b5e71-0870-42ed-bd1a-302562ecaa8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:19.920059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.920016 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-kube-api-access-nhxwx" pod "b99b5e71-0870-42ed-bd1a-302562ecaa8c" (UID: "b99b5e71-0870-42ed-bd1a-302562ecaa8c"). InnerVolumeSpecName "kube-api-access-nhxwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:19.931503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.931475 4456 generic.go:356] "Generic (PLEG): container finished" podID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerID="d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525" exitCode=0 Oct 31 03:19:19.957105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.931726 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" Oct 31 03:19:19.957105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.932508 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" event={"ID":"b99b5e71-0870-42ed-bd1a-302562ecaa8c","Type":"ContainerDied","Data":"d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525"} Oct 31 03:19:19.957105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.943723 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb" event={"ID":"b99b5e71-0870-42ed-bd1a-302562ecaa8c","Type":"ContainerDied","Data":"72d5af275b0eca7a9efd0c6d1add010288904d200ca17e694e4cb2fa2cd0d37f"} Oct 31 03:19:19.957105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.943760 4456 scope.go:122] "RemoveContainer" containerID="d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525" Oct 31 03:19:19.957105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.949557 4456 generic.go:356] "Generic (PLEG): container finished" podID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerID="428aabd3bda49291197b74e2dc70a3ecd615dfcfc45f3734d342995c02464a38" exitCode=0 Oct 31 03:19:19.957105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.950590 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" event={"ID":"635cf9bc-77c2-4809-9e6a-1f9d331431bb","Type":"ContainerDied","Data":"428aabd3bda49291197b74e2dc70a3ecd615dfcfc45f3734d342995c02464a38"} Oct 31 03:19:19.971435 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.971400 4456 scope.go:122] "RemoveContainer" containerID="d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525" Oct 31 03:19:19.971749 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:19:19.971729 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525\": container with ID starting with d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525 not found: ID does not exist" containerID="d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525" Oct 31 03:19:19.971832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:19.971759 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525"} err="failed to get container status \"d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525\": rpc error: code = NotFound desc = could not find container \"d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525\": container with ID starting with d0b5ddc8f4849f39f1742919cc778698d6c895505aa4a0d61a54c84a36bfb525 not found: ID does not exist" Oct 31 03:19:20.011847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.010658 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-socket\") pod \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\" (UID: \"b99b5e71-0870-42ed-bd1a-302562ecaa8c\") " Oct 31 03:19:20.011847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.011525 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-socket" pod "b99b5e71-0870-42ed-bd1a-302562ecaa8c" (UID: "b99b5e71-0870-42ed-bd1a-302562ecaa8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:20.011847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.011689 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.011847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.011711 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.011847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.011726 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b5e71-0870-42ed-bd1a-302562ecaa8c-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.011847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.011744 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99b5e71-0870-42ed-bd1a-302562ecaa8c-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.011847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.011759 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhxwx\" (UniqueName: \"kubernetes.io/projected/b99b5e71-0870-42ed-bd1a-302562ecaa8c-kube-api-access-nhxwx\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.275487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.275364 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb"] Oct 31 03:19:20.283722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.282945 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-glgjb"] Oct 31 03:19:20.530172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.530091 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" path="/var/lib/kubelet/pods/b99b5e71-0870-42ed-bd1a-302562ecaa8c/volumes" Oct 31 03:19:20.653393 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.653362 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:19:20.814952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.814865 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-credential-socket\") pod \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " Oct 31 03:19:20.815115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.814967 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-certs\") pod \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " Oct 31 03:19:20.815115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815006 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-kube-api-access-9j592\" (UniqueName: \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-kube-api-access-9j592\") pod \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " Oct 31 03:19:20.815201 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815113 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-socket\") pod \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " Oct 31 03:19:20.815244 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815228 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-podinfo\") pod \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " Oct 31 03:19:20.815321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815286 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-credential-socket" pod "635cf9bc-77c2-4809-9e6a-1f9d331431bb" (UID: "635cf9bc-77c2-4809-9e6a-1f9d331431bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:20.815354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815329 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-certs" pod "635cf9bc-77c2-4809-9e6a-1f9d331431bb" (UID: "635cf9bc-77c2-4809-9e6a-1f9d331431bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:20.815391 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815362 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-envoy\") pod \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " Oct 31 03:19:20.815446 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815408 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istiod-ca-cert\") pod \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " Oct 31 03:19:20.815502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815447 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-data\") pod \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " Oct 31 03:19:20.815502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815474 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-token\" (UniqueName: \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-token\") pod \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\" (UID: \"635cf9bc-77c2-4809-9e6a-1f9d331431bb\") " Oct 31 03:19:20.815632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815506 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-socket" pod "635cf9bc-77c2-4809-9e6a-1f9d331431bb" (UID: "635cf9bc-77c2-4809-9e6a-1f9d331431bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:20.816124 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815820 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istiod-ca-cert" pod "635cf9bc-77c2-4809-9e6a-1f9d331431bb" (UID: "635cf9bc-77c2-4809-9e6a-1f9d331431bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:20.816124 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.815841 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-data" pod "635cf9bc-77c2-4809-9e6a-1f9d331431bb" (UID: "635cf9bc-77c2-4809-9e6a-1f9d331431bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:20.816400 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.816377 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.816479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.816405 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.816479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.816422 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.816479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.816437 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.816479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.816450 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.817843 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.817801 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-envoy" pod "635cf9bc-77c2-4809-9e6a-1f9d331431bb" (UID: "635cf9bc-77c2-4809-9e6a-1f9d331431bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:20.818146 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.818121 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-podinfo" pod "635cf9bc-77c2-4809-9e6a-1f9d331431bb" (UID: "635cf9bc-77c2-4809-9e6a-1f9d331431bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:20.818664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.818642 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-kube-api-access-9j592" pod "635cf9bc-77c2-4809-9e6a-1f9d331431bb" (UID: "635cf9bc-77c2-4809-9e6a-1f9d331431bb"). InnerVolumeSpecName "kube-api-access-9j592". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:20.818744 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.818677 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-token" pod "635cf9bc-77c2-4809-9e6a-1f9d331431bb" (UID: "635cf9bc-77c2-4809-9e6a-1f9d331431bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:20.918568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.918451 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.918568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.918492 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.918568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.918510 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j592\" (UniqueName: \"kubernetes.io/projected/635cf9bc-77c2-4809-9e6a-1f9d331431bb-kube-api-access-9j592\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.918568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.918526 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635cf9bc-77c2-4809-9e6a-1f9d331431bb-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:20.966691 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.966647 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" event={"ID":"635cf9bc-77c2-4809-9e6a-1f9d331431bb","Type":"ContainerDied","Data":"b083eddd105a225b84a02f3caf679158e2895fff7ce97d7378222a148b808950"} Oct 31 03:19:20.966881 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.966729 4456 scope.go:122] "RemoveContainer" containerID="428aabd3bda49291197b74e2dc70a3ecd615dfcfc45f3734d342995c02464a38" Oct 31 03:19:20.966934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:20.966894 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5" Oct 31 03:19:21.029540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:21.028152 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5"] Oct 31 03:19:21.045218 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:21.045177 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-mpnw5"] Oct 31 03:19:22.444023 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:22.443977 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" path="/var/lib/kubelet/pods/635cf9bc-77c2-4809-9e6a-1f9d331431bb/volumes" Oct 31 03:19:23.293338 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.293302 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp"] Oct 31 03:19:23.294065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.294025 4456 cpu_manager.go:414] "containerMap: removing container" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" Oct 31 03:19:23.294065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.294054 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" Oct 31 03:19:23.294149 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.294081 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" Oct 31 03:19:23.294149 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.294088 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" Oct 31 03:19:23.294209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.294191 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="635cf9bc-77c2-4809-9e6a-1f9d331431bb" containerName="istio-proxy" Oct 31 03:19:23.294209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.294201 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99b5e71-0870-42ed-bd1a-302562ecaa8c" containerName="istio-proxy" Oct 31 03:19:23.299369 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.299344 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.310413 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.310312 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:19:23.310413 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.310369 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:19:23.314008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.312102 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp"] Oct 31 03:19:23.461066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.461016 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.461536 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.461389 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d7wc\" (UniqueName: \"kubernetes.io/projected/8e38aebf-2b42-486d-8650-f9ca54007d3e-kube-api-access-5d7wc\") pod \"maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.461536 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.461425 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.568103 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.565224 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d7wc\" (UniqueName: \"kubernetes.io/projected/8e38aebf-2b42-486d-8650-f9ca54007d3e-kube-api-access-5d7wc\") pod \"maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.568103 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.565283 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.568103 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.565463 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.572583 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.569525 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.574595 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.574567 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.583531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.583505 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d7wc\" (UniqueName: \"kubernetes.io/projected/8e38aebf-2b42-486d-8650-f9ca54007d3e-kube-api-access-5d7wc\") pod \"maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.627376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.627342 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:23.922203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:23.922171 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp"] Oct 31 03:19:23.923638 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:19:23.923612 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e38aebf_2b42_486d_8650_f9ca54007d3e.slice/crio-53317e22fa130ccbf5bd574fb3773e96939bf05227a326cb629a6216c1f9d9c7 WatchSource:0}: Error finding container 53317e22fa130ccbf5bd574fb3773e96939bf05227a326cb629a6216c1f9d9c7: Status 404 returned error can't find the container with id 53317e22fa130ccbf5bd574fb3773e96939bf05227a326cb629a6216c1f9d9c7 Oct 31 03:19:24.001899 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:24.001864 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" event={"ID":"8e38aebf-2b42-486d-8650-f9ca54007d3e","Type":"ContainerStarted","Data":"53317e22fa130ccbf5bd574fb3773e96939bf05227a326cb629a6216c1f9d9c7"} Oct 31 03:19:25.025390 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:25.025357 4456 generic.go:356] "Generic (PLEG): container finished" podID="8e38aebf-2b42-486d-8650-f9ca54007d3e" containerID="c2f8ebd5f6a4ecaa3b7b831d05d195d468dee75fdebde4048c68c81051b2d4ae" exitCode=0 Oct 31 03:19:25.025765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:25.025415 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" event={"ID":"8e38aebf-2b42-486d-8650-f9ca54007d3e","Type":"ContainerDied","Data":"c2f8ebd5f6a4ecaa3b7b831d05d195d468dee75fdebde4048c68c81051b2d4ae"} Oct 31 03:19:26.204850 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:26.204818 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:26.310115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:26.309984 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-openshift-ca\") pod \"8e38aebf-2b42-486d-8650-f9ca54007d3e\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " Oct 31 03:19:26.310274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:26.310186 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-xks-ca\") pod \"8e38aebf-2b42-486d-8650-f9ca54007d3e\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " Oct 31 03:19:26.310274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:26.310226 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e38aebf-2b42-486d-8650-f9ca54007d3e-kube-api-access-5d7wc\" (UniqueName: \"kubernetes.io/projected/8e38aebf-2b42-486d-8650-f9ca54007d3e-kube-api-access-5d7wc\") pod \"8e38aebf-2b42-486d-8650-f9ca54007d3e\" (UID: \"8e38aebf-2b42-486d-8650-f9ca54007d3e\") " Oct 31 03:19:26.310453 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:26.310418 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-openshift-ca" pod "8e38aebf-2b42-486d-8650-f9ca54007d3e" (UID: "8e38aebf-2b42-486d-8650-f9ca54007d3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:26.310945 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:26.310920 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:26.312322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:26.312290 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-xks-ca" pod "8e38aebf-2b42-486d-8650-f9ca54007d3e" (UID: "8e38aebf-2b42-486d-8650-f9ca54007d3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:26.312508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:26.312484 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e38aebf-2b42-486d-8650-f9ca54007d3e-kube-api-access-5d7wc" pod "8e38aebf-2b42-486d-8650-f9ca54007d3e" (UID: "8e38aebf-2b42-486d-8650-f9ca54007d3e"). InnerVolumeSpecName "kube-api-access-5d7wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:26.415063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:26.414536 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e38aebf-2b42-486d-8650-f9ca54007d3e-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:26.415063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:26.414579 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d7wc\" (UniqueName: \"kubernetes.io/projected/8e38aebf-2b42-486d-8650-f9ca54007d3e-kube-api-access-5d7wc\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:27.047865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:27.047833 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" Oct 31 03:19:27.048121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:27.047833 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp" event={"ID":"8e38aebf-2b42-486d-8650-f9ca54007d3e","Type":"ContainerDied","Data":"53317e22fa130ccbf5bd574fb3773e96939bf05227a326cb629a6216c1f9d9c7"} Oct 31 03:19:27.048121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:27.047951 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53317e22fa130ccbf5bd574fb3773e96939bf05227a326cb629a6216c1f9d9c7" Oct 31 03:19:27.976206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:27.976135 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp"] Oct 31 03:19:29.403154 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:29.403122 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt"] Oct 31 03:19:29.408405 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:29.407653 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerName="istio-proxy" containerID="cri-o://f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1" gracePeriod=30 Oct 31 03:19:29.412123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:29.412098 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g"] Oct 31 03:19:29.412503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:29.412477 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" containerID="cri-o://7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93" gracePeriod=30 Oct 31 03:19:29.426616 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:29.426589 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5"] Oct 31 03:19:29.427030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:29.427005 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" containerID="cri-o://549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1" gracePeriod=30 Oct 31 03:19:29.954393 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:29.954361 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:19:32.178086 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:19:32.175627 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:19:32.207969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.207929 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4"] Oct 31 03:19:32.208859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.208836 4456 cpu_manager.go:414] "containerMap: removing container" podUID="8e38aebf-2b42-486d-8650-f9ca54007d3e" containerName="revoke-keys" Oct 31 03:19:32.208859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.208855 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e38aebf-2b42-486d-8650-f9ca54007d3e" containerName="revoke-keys" Oct 31 03:19:32.209109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.208958 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e38aebf-2b42-486d-8650-f9ca54007d3e" containerName="revoke-keys" Oct 31 03:19:32.213340 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.213107 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:32.220595 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.220567 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:19:32.220595 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.220579 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:19:32.221322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.221301 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4"] Oct 31 03:19:32.296083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.296023 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:32.296336 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.296310 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:32.296388 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.296357 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkpzd\" (UniqueName: \"kubernetes.io/projected/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-kube-api-access-fkpzd\") pod \"maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:32.399054 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.398672 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:32.400093 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.400064 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:32.401062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.401024 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkpzd\" (UniqueName: \"kubernetes.io/projected/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-kube-api-access-fkpzd\") pod \"maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:32.401362 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.400973 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:32.413420 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.413391 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:32.422364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.422337 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkpzd\" (UniqueName: \"kubernetes.io/projected/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-kube-api-access-fkpzd\") pod \"maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:32.563110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:32.553353 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:33.052840 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:19:33.052810 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff69a1c7_9b7e_4486_ab3f_1784b34e0e58.slice/crio-afc00aaa9904bc10dc40281e30600c7509f9c4bec497d25270e1242d9a9713f4 WatchSource:0}: Error finding container afc00aaa9904bc10dc40281e30600c7509f9c4bec497d25270e1242d9a9713f4: Status 404 returned error can't find the container with id afc00aaa9904bc10dc40281e30600c7509f9c4bec497d25270e1242d9a9713f4 Oct 31 03:19:33.057478 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:33.057450 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4"] Oct 31 03:19:33.112564 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:33.112534 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" event={"ID":"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58","Type":"ContainerStarted","Data":"afc00aaa9904bc10dc40281e30600c7509f9c4bec497d25270e1242d9a9713f4"} Oct 31 03:19:34.191501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.191465 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" event={"ID":"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58","Type":"ContainerStarted","Data":"da081583d8d125f724b90608dd2a9d7bbc2defdcaa9198bd3cfaab46b79e6a17"} Oct 31 03:19:34.218822 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.218721 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" podStartSLOduration=2.218699948 podStartE2EDuration="2.218699948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:34.2164422 +0000 UTC m=+1455.896968327" watchObservedRunningTime="2026-10-31 03:19:34.218699948 +0000 UTC m=+1455.899226076" Oct 31 03:19:34.495315 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:19:34.495278 4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711922a0_faeb_4427_87d6_c71c3e3ed2e9.slice/crio-7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:19:34.842731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.842698 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:19:34.879778 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.878996 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:19:34.938125 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.938094 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:19:34.995272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995176 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-token\" (UniqueName: \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-token\") pod \"e9354b4b-c613-48a1-aebf-f2eed6be09da\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " Oct 31 03:19:34.995272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995248 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-podinfo\") pod \"e9354b4b-c613-48a1-aebf-f2eed6be09da\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " Oct 31 03:19:34.995501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995281 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-token\" (UniqueName: \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-token\") pod \"470600d7-dfd0-4741-9cdf-a840d77d5beb\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " Oct 31 03:19:34.995501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995330 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istiod-ca-cert\") pod \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " Oct 31 03:19:34.995501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995372 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-envoy\") pod \"470600d7-dfd0-4741-9cdf-a840d77d5beb\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " Oct 31 03:19:34.995501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995420 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-kube-api-access-bsksk\" (UniqueName: \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-kube-api-access-bsksk\") pod \"e9354b4b-c613-48a1-aebf-f2eed6be09da\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " Oct 31 03:19:34.995501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995487 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-envoy\") pod \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " Oct 31 03:19:34.995761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995534 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-credential-socket\") pod \"470600d7-dfd0-4741-9cdf-a840d77d5beb\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " Oct 31 03:19:34.995761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995574 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9354b4b-c613-48a1-aebf-f2eed6be09da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9354b4b-c613-48a1-aebf-f2eed6be09da-istiod-ca-cert\") pod \"e9354b4b-c613-48a1-aebf-f2eed6be09da\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " Oct 31 03:19:34.995761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995616 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-certs\") pod \"470600d7-dfd0-4741-9cdf-a840d77d5beb\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " Oct 31 03:19:34.995761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995649 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-podinfo\") pod \"470600d7-dfd0-4741-9cdf-a840d77d5beb\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " Oct 31 03:19:34.995761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995678 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-credential-socket\") pod \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " Oct 31 03:19:34.995761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995723 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-kube-api-access-6gv72\" (UniqueName: \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-kube-api-access-6gv72\") pod \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " Oct 31 03:19:34.996141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995780 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-token\" (UniqueName: \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-token\") pod \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " Oct 31 03:19:34.996141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995822 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-kube-api-access-ptl2k\" (UniqueName: \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-kube-api-access-ptl2k\") pod \"470600d7-dfd0-4741-9cdf-a840d77d5beb\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " Oct 31 03:19:34.996141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995870 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-certs\") pod \"e9354b4b-c613-48a1-aebf-f2eed6be09da\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " Oct 31 03:19:34.996141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995962 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-socket\") pod \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " Oct 31 03:19:34.996141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.995999 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/470600d7-dfd0-4741-9cdf-a840d77d5beb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470600d7-dfd0-4741-9cdf-a840d77d5beb-istiod-ca-cert\") pod \"470600d7-dfd0-4741-9cdf-a840d77d5beb\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " Oct 31 03:19:34.996141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.996028 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-credential-socket\") pod \"e9354b4b-c613-48a1-aebf-f2eed6be09da\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " Oct 31 03:19:34.996141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.996104 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-data\") pod \"470600d7-dfd0-4741-9cdf-a840d77d5beb\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " Oct 31 03:19:34.996141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.996141 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-socket\") pod \"e9354b4b-c613-48a1-aebf-f2eed6be09da\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " Oct 31 03:19:34.996560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.996193 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-socket\") pod \"470600d7-dfd0-4741-9cdf-a840d77d5beb\" (UID: \"470600d7-dfd0-4741-9cdf-a840d77d5beb\") " Oct 31 03:19:34.996560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.996244 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-data\") pod \"e9354b4b-c613-48a1-aebf-f2eed6be09da\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " Oct 31 03:19:34.996560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.996296 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-envoy\") pod \"e9354b4b-c613-48a1-aebf-f2eed6be09da\" (UID: \"e9354b4b-c613-48a1-aebf-f2eed6be09da\") " Oct 31 03:19:34.996560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.996351 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-data\") pod \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " Oct 31 03:19:34.996560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.996381 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-certs\") pod \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " Oct 31 03:19:34.996560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.996413 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-podinfo\") pod \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\" (UID: \"711922a0-faeb-4427-87d6-c71c3e3ed2e9\") " Oct 31 03:19:34.999408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.997587 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-data" pod "e9354b4b-c613-48a1-aebf-f2eed6be09da" (UID: "e9354b4b-c613-48a1-aebf-f2eed6be09da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.999408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.997629 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-certs" pod "711922a0-faeb-4427-87d6-c71c3e3ed2e9" (UID: "711922a0-faeb-4427-87d6-c71c3e3ed2e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.999408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.997781 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-data" pod "711922a0-faeb-4427-87d6-c71c3e3ed2e9" (UID: "711922a0-faeb-4427-87d6-c71c3e3ed2e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.999408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.997950 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9354b4b-c613-48a1-aebf-f2eed6be09da-istiod-ca-cert" pod "e9354b4b-c613-48a1-aebf-f2eed6be09da" (UID: "e9354b4b-c613-48a1-aebf-f2eed6be09da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:34.999408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.997980 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-data" pod "470600d7-dfd0-4741-9cdf-a840d77d5beb" (UID: "470600d7-dfd0-4741-9cdf-a840d77d5beb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.999408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.998292 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-socket" pod "470600d7-dfd0-4741-9cdf-a840d77d5beb" (UID: "470600d7-dfd0-4741-9cdf-a840d77d5beb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.999408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.998589 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-socket" pod "e9354b4b-c613-48a1-aebf-f2eed6be09da" (UID: "e9354b4b-c613-48a1-aebf-f2eed6be09da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.999408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.998859 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-credential-socket" pod "470600d7-dfd0-4741-9cdf-a840d77d5beb" (UID: "470600d7-dfd0-4741-9cdf-a840d77d5beb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.999408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:34.999142 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istiod-ca-cert" pod "711922a0-faeb-4427-87d6-c71c3e3ed2e9" (UID: "711922a0-faeb-4427-87d6-c71c3e3ed2e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:35.000120 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.000074 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-envoy" pod "e9354b4b-c613-48a1-aebf-f2eed6be09da" (UID: "e9354b4b-c613-48a1-aebf-f2eed6be09da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.000577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.000121 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-kube-api-access-6gv72" pod "711922a0-faeb-4427-87d6-c71c3e3ed2e9" (UID: "711922a0-faeb-4427-87d6-c71c3e3ed2e9"). InnerVolumeSpecName "kube-api-access-6gv72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.000577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.000276 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-envoy" pod "711922a0-faeb-4427-87d6-c71c3e3ed2e9" (UID: "711922a0-faeb-4427-87d6-c71c3e3ed2e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.000577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.000380 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-certs" pod "470600d7-dfd0-4741-9cdf-a840d77d5beb" (UID: "470600d7-dfd0-4741-9cdf-a840d77d5beb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.000792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.000711 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-credential-socket" pod "711922a0-faeb-4427-87d6-c71c3e3ed2e9" (UID: "711922a0-faeb-4427-87d6-c71c3e3ed2e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.001262 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.000962 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-token" pod "711922a0-faeb-4427-87d6-c71c3e3ed2e9" (UID: "711922a0-faeb-4427-87d6-c71c3e3ed2e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.001262 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.001176 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-kube-api-access-bsksk" pod "e9354b4b-c613-48a1-aebf-f2eed6be09da" (UID: "e9354b4b-c613-48a1-aebf-f2eed6be09da"). InnerVolumeSpecName "kube-api-access-bsksk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.001519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.001492 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-podinfo" pod "711922a0-faeb-4427-87d6-c71c3e3ed2e9" (UID: "711922a0-faeb-4427-87d6-c71c3e3ed2e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:35.001600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.001493 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-socket" pod "711922a0-faeb-4427-87d6-c71c3e3ed2e9" (UID: "711922a0-faeb-4427-87d6-c71c3e3ed2e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.001784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.001692 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-certs" pod "e9354b4b-c613-48a1-aebf-f2eed6be09da" (UID: "e9354b4b-c613-48a1-aebf-f2eed6be09da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.001784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.001735 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-envoy" pod "470600d7-dfd0-4741-9cdf-a840d77d5beb" (UID: "470600d7-dfd0-4741-9cdf-a840d77d5beb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.001963 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.001863 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/470600d7-dfd0-4741-9cdf-a840d77d5beb-istiod-ca-cert" pod "470600d7-dfd0-4741-9cdf-a840d77d5beb" (UID: "470600d7-dfd0-4741-9cdf-a840d77d5beb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:35.001963 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.001905 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-credential-socket" pod "e9354b4b-c613-48a1-aebf-f2eed6be09da" (UID: "e9354b4b-c613-48a1-aebf-f2eed6be09da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.002376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.002346 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-token" pod "e9354b4b-c613-48a1-aebf-f2eed6be09da" (UID: "e9354b4b-c613-48a1-aebf-f2eed6be09da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.002905 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.002872 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-podinfo" pod "e9354b4b-c613-48a1-aebf-f2eed6be09da" (UID: "e9354b4b-c613-48a1-aebf-f2eed6be09da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:35.002997 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.002965 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-token" pod "470600d7-dfd0-4741-9cdf-a840d77d5beb" (UID: "470600d7-dfd0-4741-9cdf-a840d77d5beb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.003434 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.003409 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-podinfo" pod "470600d7-dfd0-4741-9cdf-a840d77d5beb" (UID: "470600d7-dfd0-4741-9cdf-a840d77d5beb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:35.003772 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.003737 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-kube-api-access-ptl2k" pod "470600d7-dfd0-4741-9cdf-a840d77d5beb" (UID: "470600d7-dfd0-4741-9cdf-a840d77d5beb"). InnerVolumeSpecName "kube-api-access-ptl2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.098168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098132 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098162 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098172 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098182 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098190 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098202 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsksk\" (UniqueName: \"kubernetes.io/projected/e9354b4b-c613-48a1-aebf-f2eed6be09da-kube-api-access-bsksk\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098212 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098220 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098229 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9354b4b-c613-48a1-aebf-f2eed6be09da-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098237 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098247 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098255 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098263 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gv72\" (UniqueName: \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-kube-api-access-6gv72\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098271 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098280 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptl2k\" (UniqueName: \"kubernetes.io/projected/470600d7-dfd0-4741-9cdf-a840d77d5beb-kube-api-access-ptl2k\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098288 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098300 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098309 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470600d7-dfd0-4741-9cdf-a840d77d5beb-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098317 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098326 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098334 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098342 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470600d7-dfd0-4741-9cdf-a840d77d5beb-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098350 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098358 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9354b4b-c613-48a1-aebf-f2eed6be09da-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098366 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098376 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711922a0-faeb-4427-87d6-c71c3e3ed2e9-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.098397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.098397 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711922a0-faeb-4427-87d6-c71c3e3ed2e9-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.203328 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.203292 4456 generic.go:356] "Generic (PLEG): container finished" podID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerID="f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1" exitCode=0 Oct 31 03:19:35.203786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.203357 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" Oct 31 03:19:35.203786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.203379 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" event={"ID":"470600d7-dfd0-4741-9cdf-a840d77d5beb","Type":"ContainerDied","Data":"f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1"} Oct 31 03:19:35.203786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.203454 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt" event={"ID":"470600d7-dfd0-4741-9cdf-a840d77d5beb","Type":"ContainerDied","Data":"12d0842043ebdd5db19ff51a2a75f79aac54867a4cb5b3a6ac0e3eaa9a1d222d"} Oct 31 03:19:35.203786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.203485 4456 scope.go:122] "RemoveContainer" containerID="f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1" Oct 31 03:19:35.205627 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.205599 4456 generic.go:356] "Generic (PLEG): container finished" podID="ff69a1c7-9b7e-4486-ab3f-1784b34e0e58" containerID="da081583d8d125f724b90608dd2a9d7bbc2defdcaa9198bd3cfaab46b79e6a17" exitCode=0 Oct 31 03:19:35.205751 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.205696 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" event={"ID":"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58","Type":"ContainerDied","Data":"da081583d8d125f724b90608dd2a9d7bbc2defdcaa9198bd3cfaab46b79e6a17"} Oct 31 03:19:35.211876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.208006 4456 generic.go:356] "Generic (PLEG): container finished" podID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerID="549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1" exitCode=0 Oct 31 03:19:35.211876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.208528 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" Oct 31 03:19:35.211876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.208946 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" event={"ID":"e9354b4b-c613-48a1-aebf-f2eed6be09da","Type":"ContainerDied","Data":"549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1"} Oct 31 03:19:35.211876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.208991 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5" event={"ID":"e9354b4b-c613-48a1-aebf-f2eed6be09da","Type":"ContainerDied","Data":"05efd6266c1c1772b5f62ffc74605bc1b7434ddd38468ba3c175db7691362997"} Oct 31 03:19:35.211876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.211712 4456 generic.go:356] "Generic (PLEG): container finished" podID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerID="7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93" exitCode=0 Oct 31 03:19:35.211876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.211767 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" Oct 31 03:19:35.211876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.211844 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" event={"ID":"711922a0-faeb-4427-87d6-c71c3e3ed2e9","Type":"ContainerDied","Data":"7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93"} Oct 31 03:19:35.212237 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.211883 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g" event={"ID":"711922a0-faeb-4427-87d6-c71c3e3ed2e9","Type":"ContainerDied","Data":"03853c1ce3c664b051e69fb4b968627f455c5c97c2fa8484b3b162a6a0fa477a"} Oct 31 03:19:35.221337 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.221316 4456 scope.go:122] "RemoveContainer" containerID="f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1" Oct 31 03:19:35.221682 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:19:35.221649 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1\": container with ID starting with f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1 not found: ID does not exist" containerID="f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1" Oct 31 03:19:35.221805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.221693 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1"} err="failed to get container status \"f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1\": rpc error: code = NotFound desc = could not find container \"f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1\": container with ID starting with f2e821987c99f33803c26cc6b6720c13738b62c41f018716d0f338c9b94bcde1 not found: ID does not exist" Oct 31 03:19:35.221805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.221717 4456 scope.go:122] "RemoveContainer" containerID="549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1" Oct 31 03:19:35.237232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.237204 4456 scope.go:122] "RemoveContainer" containerID="549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1" Oct 31 03:19:35.237737 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:19:35.237709 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1\": container with ID starting with 549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1 not found: ID does not exist" containerID="549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1" Oct 31 03:19:35.237835 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.237748 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1"} err="failed to get container status \"549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1\": rpc error: code = NotFound desc = could not find container \"549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1\": container with ID starting with 549f9ca6f356dbb60e66253040c7de433338222071dcbdeeff2474f3a0afcbe1 not found: ID does not exist" Oct 31 03:19:35.237835 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.237771 4456 scope.go:122] "RemoveContainer" containerID="7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93" Oct 31 03:19:35.249259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.249239 4456 scope.go:122] "RemoveContainer" containerID="7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93" Oct 31 03:19:35.249580 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:19:35.249553 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93\": container with ID starting with 7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93 not found: ID does not exist" containerID="7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93" Oct 31 03:19:35.249652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.249593 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93"} err="failed to get container status \"7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93\": rpc error: code = NotFound desc = could not find container \"7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93\": container with ID starting with 7e6e41866e4070d36417c1319669852280883a00b295f8cd1cebbd8af916ec93 not found: ID does not exist" Oct 31 03:19:35.269232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.269202 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g"] Oct 31 03:19:35.274844 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.274818 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-lrt7g"] Oct 31 03:19:35.289508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.289479 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5"] Oct 31 03:19:35.295007 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.294980 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-z29x5"] Oct 31 03:19:35.305658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.305628 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt"] Oct 31 03:19:35.310030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:35.310005 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-nkldt"] Oct 31 03:19:36.442098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.442060 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" path="/var/lib/kubelet/pods/470600d7-dfd0-4741-9cdf-a840d77d5beb/volumes" Oct 31 03:19:36.442836 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.442815 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" path="/var/lib/kubelet/pods/711922a0-faeb-4427-87d6-c71c3e3ed2e9/volumes" Oct 31 03:19:36.443448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.443431 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:36.443525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.443498 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" path="/var/lib/kubelet/pods/e9354b4b-c613-48a1-aebf-f2eed6be09da/volumes" Oct 31 03:19:36.521411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.521334 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-openshift-ca\") pod \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " Oct 31 03:19:36.521581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.521480 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-kube-api-access-fkpzd\" (UniqueName: \"kubernetes.io/projected/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-kube-api-access-fkpzd\") pod \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " Oct 31 03:19:36.521581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.521527 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-xks-ca\") pod \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\" (UID: \"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58\") " Oct 31 03:19:36.521774 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.521745 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-openshift-ca" pod "ff69a1c7-9b7e-4486-ab3f-1784b34e0e58" (UID: "ff69a1c7-9b7e-4486-ab3f-1784b34e0e58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:36.522702 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.522662 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:36.524444 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.524422 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-xks-ca" pod "ff69a1c7-9b7e-4486-ab3f-1784b34e0e58" (UID: "ff69a1c7-9b7e-4486-ab3f-1784b34e0e58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:36.525711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.525686 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-kube-api-access-fkpzd" pod "ff69a1c7-9b7e-4486-ab3f-1784b34e0e58" (UID: "ff69a1c7-9b7e-4486-ab3f-1784b34e0e58"). InnerVolumeSpecName "kube-api-access-fkpzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:36.624363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.624331 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkpzd\" (UniqueName: \"kubernetes.io/projected/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-kube-api-access-fkpzd\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:36.624363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:36.624362 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:37.234168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.234134 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" Oct 31 03:19:37.234319 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.234129 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4" event={"ID":"ff69a1c7-9b7e-4486-ab3f-1784b34e0e58","Type":"ContainerDied","Data":"afc00aaa9904bc10dc40281e30600c7509f9c4bec497d25270e1242d9a9713f4"} Oct 31 03:19:37.234319 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.234263 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afc00aaa9904bc10dc40281e30600c7509f9c4bec497d25270e1242d9a9713f4" Oct 31 03:19:37.925322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.925289 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm"] Oct 31 03:19:37.926020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926003 4456 cpu_manager.go:414] "containerMap: removing container" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" Oct 31 03:19:37.926020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926018 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" Oct 31 03:19:37.926196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926031 4456 cpu_manager.go:414] "containerMap: removing container" podUID="ff69a1c7-9b7e-4486-ab3f-1784b34e0e58" containerName="revoke-keys" Oct 31 03:19:37.926196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926051 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff69a1c7-9b7e-4486-ab3f-1784b34e0e58" containerName="revoke-keys" Oct 31 03:19:37.926196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926063 4456 cpu_manager.go:414] "containerMap: removing container" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerName="istio-proxy" Oct 31 03:19:37.926196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926068 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerName="istio-proxy" Oct 31 03:19:37.926196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926075 4456 cpu_manager.go:414] "containerMap: removing container" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" Oct 31 03:19:37.926196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926079 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" Oct 31 03:19:37.926196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926188 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="711922a0-faeb-4427-87d6-c71c3e3ed2e9" containerName="istio-proxy" Oct 31 03:19:37.926196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926197 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff69a1c7-9b7e-4486-ab3f-1784b34e0e58" containerName="revoke-keys" Oct 31 03:19:37.926430 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926204 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="470600d7-dfd0-4741-9cdf-a840d77d5beb" containerName="istio-proxy" Oct 31 03:19:37.926430 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.926210 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9354b4b-c613-48a1-aebf-f2eed6be09da" containerName="istio-proxy" Oct 31 03:19:37.931360 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.931339 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:19:37.941267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.941237 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm"] Oct 31 03:19:37.941662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.941639 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fccff5\"" Oct 31 03:19:37.941771 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.941676 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fccff5-dockercfg-j49q5\"" Oct 31 03:19:37.957698 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.957671 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc"] Oct 31 03:19:37.962378 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.962352 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:19:37.970399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:37.970297 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc"] Oct 31 03:19:38.062076 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.045978 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm\" (UID: \"854d04e7-2033-42e6-aeea-1d1fa65ecb7e\") " pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:19:38.062076 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.046083 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp6d9\" (UniqueName: \"kubernetes.io/projected/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-kube-api-access-lp6d9\") pod \"payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm\" (UID: \"854d04e7-2033-42e6-aeea-1d1fa65ecb7e\") " pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:19:38.062076 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.046182 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2cq6\" (UniqueName: \"kubernetes.io/projected/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-kube-api-access-b2cq6\") pod \"payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc\" (UID: \"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:19:38.062076 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.046265 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc\" (UID: \"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:19:38.153967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.153922 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp6d9\" (UniqueName: \"kubernetes.io/projected/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-kube-api-access-lp6d9\") pod \"payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm\" (UID: \"854d04e7-2033-42e6-aeea-1d1fa65ecb7e\") " pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:19:38.154176 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.154062 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2cq6\" (UniqueName: \"kubernetes.io/projected/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-kube-api-access-b2cq6\") pod \"payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc\" (UID: \"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:19:38.154176 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.154146 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc\" (UID: \"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:19:38.154584 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.154362 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm\" (UID: \"854d04e7-2033-42e6-aeea-1d1fa65ecb7e\") " pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:19:38.155410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.155382 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm\" (UID: \"854d04e7-2033-42e6-aeea-1d1fa65ecb7e\") " pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:19:38.168060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.165511 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc\" (UID: \"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:19:38.173856 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.173827 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2cq6\" (UniqueName: \"kubernetes.io/projected/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-kube-api-access-b2cq6\") pod \"payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc\" (UID: \"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:19:38.190134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.181696 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp6d9\" (UniqueName: \"kubernetes.io/projected/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-kube-api-access-lp6d9\") pod \"payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm\" (UID: \"854d04e7-2033-42e6-aeea-1d1fa65ecb7e\") " pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:19:38.258173 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.257847 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:19:38.283515 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.283474 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:19:38.636268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.636230 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm"] Oct 31 03:19:38.693659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:38.693624 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc"] Oct 31 03:19:39.258999 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:39.258958 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" event={"ID":"854d04e7-2033-42e6-aeea-1d1fa65ecb7e","Type":"ContainerStarted","Data":"b6b9796d9b3300d33b23e873ff3e943145f18c3a6ff97f6f0f90a87046e2a4b3"} Oct 31 03:19:39.259521 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:39.259081 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:19:39.259521 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:39.259112 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" event={"ID":"854d04e7-2033-42e6-aeea-1d1fa65ecb7e","Type":"ContainerStarted","Data":"03086fe0303c2b7e0514a7f63a10537443933c26f94b8e3d18c40ee3bea275ea"} Oct 31 03:19:39.260998 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:39.260973 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" event={"ID":"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e","Type":"ContainerStarted","Data":"f39b3d3ed05d4c1387026de771e325edbfe538d93641a5aa711d751b6faf6e16"} Oct 31 03:19:39.261145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:39.261017 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" event={"ID":"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e","Type":"ContainerStarted","Data":"ad864a5ac334aca19aa1b65aaf94e216a14b98f73d89d3651585030296cbe989"} Oct 31 03:19:39.261145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:39.261105 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:19:39.286904 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:39.286852 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" podStartSLOduration=2.286838479 podStartE2EDuration="2.286838479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:39.281939344 +0000 UTC m=+1460.962465472" watchObservedRunningTime="2026-10-31 03:19:39.286838479 +0000 UTC m=+1460.967364606" Oct 31 03:19:39.307456 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:39.307409 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" podStartSLOduration=2.307395742 podStartE2EDuration="2.307395742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:39.30557041 +0000 UTC m=+1460.986096537" watchObservedRunningTime="2026-10-31 03:19:39.307395742 +0000 UTC m=+1460.987921930" Oct 31 03:19:43.001739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.001689 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" containerID="cri-o://66f73d4c54771be7b70924060ef637dce1a3996cb24c001fc31f0f974360fa33" gracePeriod=45 Oct 31 03:19:43.303613 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.303577 4456 generic.go:356] "Generic (PLEG): container finished" podID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerID="66f73d4c54771be7b70924060ef637dce1a3996cb24c001fc31f0f974360fa33" exitCode=0 Oct 31 03:19:43.303792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.303655 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" event={"ID":"8fb148a9-9b6c-42ab-a831-dd992755ad73","Type":"ContainerDied","Data":"66f73d4c54771be7b70924060ef637dce1a3996cb24c001fc31f0f974360fa33"} Oct 31 03:19:43.303792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.303699 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" event={"ID":"8fb148a9-9b6c-42ab-a831-dd992755ad73","Type":"ContainerDied","Data":"9d587d5a711f019b8be3dc95f5fc5485dfbdb56808727d3b60470147dd5b99fc"} Oct 31 03:19:43.303792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.303714 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d587d5a711f019b8be3dc95f5fc5485dfbdb56808727d3b60470147dd5b99fc" Oct 31 03:19:43.309122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.309101 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:19:43.395873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.393919 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-dshm\") pod \"8fb148a9-9b6c-42ab-a831-dd992755ad73\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " Oct 31 03:19:43.395873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.393975 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-home\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-home\") pod \"8fb148a9-9b6c-42ab-a831-dd992755ad73\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " Oct 31 03:19:43.395873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.394021 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-tmp-dir\") pod \"8fb148a9-9b6c-42ab-a831-dd992755ad73\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " Oct 31 03:19:43.395873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.394268 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb148a9-9b6c-42ab-a831-dd992755ad73-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb148a9-9b6c-42ab-a831-dd992755ad73-tls-certs\") pod \"8fb148a9-9b6c-42ab-a831-dd992755ad73\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " Oct 31 03:19:43.395873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.394327 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-model-cache\") pod \"8fb148a9-9b6c-42ab-a831-dd992755ad73\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " Oct 31 03:19:43.395873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.394369 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb148a9-9b6c-42ab-a831-dd992755ad73-kube-api-access-82wbb\" (UniqueName: \"kubernetes.io/projected/8fb148a9-9b6c-42ab-a831-dd992755ad73-kube-api-access-82wbb\") pod \"8fb148a9-9b6c-42ab-a831-dd992755ad73\" (UID: \"8fb148a9-9b6c-42ab-a831-dd992755ad73\") " Oct 31 03:19:43.395873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.395543 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-tmp-dir" pod "8fb148a9-9b6c-42ab-a831-dd992755ad73" (UID: "8fb148a9-9b6c-42ab-a831-dd992755ad73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.398472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.398374 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-model-cache" pod "8fb148a9-9b6c-42ab-a831-dd992755ad73" (UID: "8fb148a9-9b6c-42ab-a831-dd992755ad73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.398472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.398430 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-dshm" pod "8fb148a9-9b6c-42ab-a831-dd992755ad73" (UID: "8fb148a9-9b6c-42ab-a831-dd992755ad73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.398472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.398443 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-home" pod "8fb148a9-9b6c-42ab-a831-dd992755ad73" (UID: "8fb148a9-9b6c-42ab-a831-dd992755ad73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.399244 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.399210 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb148a9-9b6c-42ab-a831-dd992755ad73-kube-api-access-82wbb" pod "8fb148a9-9b6c-42ab-a831-dd992755ad73" (UID: "8fb148a9-9b6c-42ab-a831-dd992755ad73"). InnerVolumeSpecName "kube-api-access-82wbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:43.400239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.400219 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb148a9-9b6c-42ab-a831-dd992755ad73-tls-certs" pod "8fb148a9-9b6c-42ab-a831-dd992755ad73" (UID: "8fb148a9-9b6c-42ab-a831-dd992755ad73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:43.496488 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.496453 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.496488 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.496488 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.496770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.496503 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb148a9-9b6c-42ab-a831-dd992755ad73-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.496770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.496516 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.496770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.496531 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82wbb\" (UniqueName: \"kubernetes.io/projected/8fb148a9-9b6c-42ab-a831-dd992755ad73-kube-api-access-82wbb\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.496770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:43.496544 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb148a9-9b6c-42ab-a831-dd992755ad73-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:44.312599 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:44.312566 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp" Oct 31 03:19:44.339838 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:44.339806 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp"] Oct 31 03:19:44.347030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:44.346999 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/premium-w0-edb9e6-kserve-557cf99995-5bpsp"] Oct 31 03:19:44.436840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:44.436806 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" path="/var/lib/kubelet/pods/8fb148a9-9b6c-42ab-a831-dd992755ad73/volumes" Oct 31 03:19:45.055993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.055952 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr"] Oct 31 03:19:45.057160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.057136 4456 cpu_manager.go:414] "containerMap: removing container" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" Oct 31 03:19:45.057160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.057159 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" Oct 31 03:19:45.057398 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.057383 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb148a9-9b6c-42ab-a831-dd992755ad73" containerName="main" Oct 31 03:19:45.067415 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.065417 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.075311 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.075262 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr"] Oct 31 03:19:45.075800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.075771 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:19:45.075800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.075771 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:19:45.097065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.096387 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd"] Oct 31 03:19:45.103295 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.103265 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.151222 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.151126 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd"] Oct 31 03:19:45.210060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.206915 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8s7b\" (UniqueName: \"kubernetes.io/projected/d54db39e-4c28-4c13-8272-63bdf94286e0-kube-api-access-w8s7b\") pod \"maas-api-revoke-sub-0ae54720-t9znr\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.210060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.207181 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ae54720-t9znr\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.210060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.207226 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ae54720-t9znr\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.345732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.318806 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02410db4-v9cvd\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.345732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.318881 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02410db4-v9cvd\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.345732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.318922 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8s7b\" (UniqueName: \"kubernetes.io/projected/d54db39e-4c28-4c13-8272-63bdf94286e0-kube-api-access-w8s7b\") pod \"maas-api-revoke-sub-0ae54720-t9znr\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.345732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.318948 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfz94\" (UniqueName: \"kubernetes.io/projected/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-kube-api-access-zfz94\") pod \"maas-api-revoke-sub-02410db4-v9cvd\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.345732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.319091 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ae54720-t9znr\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.345732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.319115 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ae54720-t9znr\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.345732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.321359 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ae54720-t9znr\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.345732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.324621 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ae54720-t9znr\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.345732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.343272 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8s7b\" (UniqueName: \"kubernetes.io/projected/d54db39e-4c28-4c13-8272-63bdf94286e0-kube-api-access-w8s7b\") pod \"maas-api-revoke-sub-0ae54720-t9znr\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.411072 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.410697 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:45.420942 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.420918 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02410db4-v9cvd\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.421044 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.420972 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02410db4-v9cvd\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.421044 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.421016 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfz94\" (UniqueName: \"kubernetes.io/projected/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-kube-api-access-zfz94\") pod \"maas-api-revoke-sub-02410db4-v9cvd\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.421805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.421779 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02410db4-v9cvd\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.423439 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.423415 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02410db4-v9cvd\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.434194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.434171 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfz94\" (UniqueName: \"kubernetes.io/projected/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-kube-api-access-zfz94\") pod \"maas-api-revoke-sub-02410db4-v9cvd\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.493544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.491342 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:45.710865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.708922 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr"] Oct 31 03:19:45.837357 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:19:45.837325 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c98d6d_570d_42ec_ae2c_8b2f7f1c3d84.slice/crio-4b11628a96f5d07ae4ad03bffa68f72a7f03f3d53d72e3d7969f7e7c1c8e727b WatchSource:0}: Error finding container 4b11628a96f5d07ae4ad03bffa68f72a7f03f3d53d72e3d7969f7e7c1c8e727b: Status 404 returned error can't find the container with id 4b11628a96f5d07ae4ad03bffa68f72a7f03f3d53d72e3d7969f7e7c1c8e727b Oct 31 03:19:45.838765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:45.838736 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd"] Oct 31 03:19:46.358406 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:46.358369 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" event={"ID":"d54db39e-4c28-4c13-8272-63bdf94286e0","Type":"ContainerStarted","Data":"9e41b1f810d4bf9162180a0f2a88f839ce2ecc6b0d26b01c568819e77fd77699"} Oct 31 03:19:46.358851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:46.358446 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" event={"ID":"d54db39e-4c28-4c13-8272-63bdf94286e0","Type":"ContainerStarted","Data":"00dc604147c71bfc918f5710b9dab8620eff5ee3190206e31171209c815932ad"} Oct 31 03:19:46.360278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:46.360249 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" event={"ID":"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84","Type":"ContainerStarted","Data":"6fd3ce937362fd40f6835c214f6324cce9a71fca752341a90a90f11a5c4a9089"} Oct 31 03:19:46.360412 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:46.360294 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" event={"ID":"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84","Type":"ContainerStarted","Data":"4b11628a96f5d07ae4ad03bffa68f72a7f03f3d53d72e3d7969f7e7c1c8e727b"} Oct 31 03:19:46.381600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:46.381551 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" podStartSLOduration=1.3815390029999999 podStartE2EDuration="1.381539003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:46.381024945 +0000 UTC m=+1468.061551073" watchObservedRunningTime="2026-10-31 03:19:46.381539003 +0000 UTC m=+1468.062065163" Oct 31 03:19:46.405734 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:46.405682 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" podStartSLOduration=1.405669292 podStartE2EDuration="1.405669292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:46.402674732 +0000 UTC m=+1468.083200860" watchObservedRunningTime="2026-10-31 03:19:46.405669292 +0000 UTC m=+1468.086195420" Oct 31 03:19:47.387792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:47.387750 4456 generic.go:356] "Generic (PLEG): container finished" podID="d54db39e-4c28-4c13-8272-63bdf94286e0" containerID="9e41b1f810d4bf9162180a0f2a88f839ce2ecc6b0d26b01c568819e77fd77699" exitCode=0 Oct 31 03:19:47.388394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:47.387855 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" event={"ID":"d54db39e-4c28-4c13-8272-63bdf94286e0","Type":"ContainerDied","Data":"9e41b1f810d4bf9162180a0f2a88f839ce2ecc6b0d26b01c568819e77fd77699"} Oct 31 03:19:47.390281 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:47.390256 4456 generic.go:356] "Generic (PLEG): container finished" podID="33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84" containerID="6fd3ce937362fd40f6835c214f6324cce9a71fca752341a90a90f11a5c4a9089" exitCode=0 Oct 31 03:19:47.390411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:47.390392 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" event={"ID":"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84","Type":"ContainerDied","Data":"6fd3ce937362fd40f6835c214f6324cce9a71fca752341a90a90f11a5c4a9089"} Oct 31 03:19:48.601982 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.601343 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:48.694344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.694292 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-xks-ca\") pod \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " Oct 31 03:19:48.694552 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.694445 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-kube-api-access-zfz94\" (UniqueName: \"kubernetes.io/projected/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-kube-api-access-zfz94\") pod \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " Oct 31 03:19:48.694552 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.694511 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-openshift-ca\") pod \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\" (UID: \"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84\") " Oct 31 03:19:48.695994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.695964 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-openshift-ca" pod "33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84" (UID: "33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:48.698526 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.698476 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-xks-ca" pod "33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84" (UID: "33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:48.702366 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.702341 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:48.703208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.703180 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-kube-api-access-zfz94" pod "33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84" (UID: "33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84"). InnerVolumeSpecName "kube-api-access-zfz94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:48.796223 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.796179 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-openshift-ca\") pod \"d54db39e-4c28-4c13-8272-63bdf94286e0\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " Oct 31 03:19:48.796392 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.796324 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-xks-ca\") pod \"d54db39e-4c28-4c13-8272-63bdf94286e0\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " Oct 31 03:19:48.796392 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.796374 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54db39e-4c28-4c13-8272-63bdf94286e0-kube-api-access-w8s7b\" (UniqueName: \"kubernetes.io/projected/d54db39e-4c28-4c13-8272-63bdf94286e0-kube-api-access-w8s7b\") pod \"d54db39e-4c28-4c13-8272-63bdf94286e0\" (UID: \"d54db39e-4c28-4c13-8272-63bdf94286e0\") " Oct 31 03:19:48.796610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.796587 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-openshift-ca" pod "d54db39e-4c28-4c13-8272-63bdf94286e0" (UID: "d54db39e-4c28-4c13-8272-63bdf94286e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:48.797164 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.797140 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:48.797256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.797169 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfz94\" (UniqueName: \"kubernetes.io/projected/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-kube-api-access-zfz94\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:48.797256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.797185 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:48.797256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.797202 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:48.798706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.798676 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54db39e-4c28-4c13-8272-63bdf94286e0-kube-api-access-w8s7b" pod "d54db39e-4c28-4c13-8272-63bdf94286e0" (UID: "d54db39e-4c28-4c13-8272-63bdf94286e0"). InnerVolumeSpecName "kube-api-access-w8s7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:48.798837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.798708 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-xks-ca" pod "d54db39e-4c28-4c13-8272-63bdf94286e0" (UID: "d54db39e-4c28-4c13-8272-63bdf94286e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:48.898261 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.898225 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d54db39e-4c28-4c13-8272-63bdf94286e0-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:48.898261 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:48.898252 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8s7b\" (UniqueName: \"kubernetes.io/projected/d54db39e-4c28-4c13-8272-63bdf94286e0-kube-api-access-w8s7b\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:49.410127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:49.410092 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" Oct 31 03:19:49.410127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:49.410118 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr" event={"ID":"d54db39e-4c28-4c13-8272-63bdf94286e0","Type":"ContainerDied","Data":"00dc604147c71bfc918f5710b9dab8620eff5ee3190206e31171209c815932ad"} Oct 31 03:19:49.410352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:49.410165 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00dc604147c71bfc918f5710b9dab8620eff5ee3190206e31171209c815932ad" Oct 31 03:19:49.411863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:49.411830 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" Oct 31 03:19:49.411863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:49.411835 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd" event={"ID":"33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84","Type":"ContainerDied","Data":"4b11628a96f5d07ae4ad03bffa68f72a7f03f3d53d72e3d7969f7e7c1c8e727b"} Oct 31 03:19:49.412110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:49.411877 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b11628a96f5d07ae4ad03bffa68f72a7f03f3d53d72e3d7969f7e7c1c8e727b" Oct 31 03:19:50.273579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:50.273546 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:19:50.274086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:50.273632 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:19:53.823499 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.823461 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v"] Oct 31 03:19:53.824497 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.824470 4456 cpu_manager.go:414] "containerMap: removing container" podUID="d54db39e-4c28-4c13-8272-63bdf94286e0" containerName="revoke-keys" Oct 31 03:19:53.824497 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.824498 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54db39e-4c28-4c13-8272-63bdf94286e0" containerName="revoke-keys" Oct 31 03:19:53.824716 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.824554 4456 cpu_manager.go:414] "containerMap: removing container" podUID="33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84" containerName="revoke-keys" Oct 31 03:19:53.824716 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.824565 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84" containerName="revoke-keys" Oct 31 03:19:53.824820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.824761 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84" containerName="revoke-keys" Oct 31 03:19:53.824820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.824781 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54db39e-4c28-4c13-8272-63bdf94286e0" containerName="revoke-keys" Oct 31 03:19:53.830092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.830069 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:53.837815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.837793 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a2bec7\"/\"openshift-service-ca.crt\"" Oct 31 03:19:53.843889 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.843857 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v"] Oct 31 03:19:53.844360 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.844335 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a2bec7\"/\"default-dockercfg-xmh74\"" Oct 31 03:19:53.844468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.844440 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a2bec7\"/\"test-model-a2bec7-kserve-self-signed-certs\"" Oct 31 03:19:53.844525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.844496 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a2bec7\"/\"kube-root-ca.crt\"" Oct 31 03:19:53.920734 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.920681 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d954ed3-9585-4249-8c87-32ee474126a7-tls-certs\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:53.920937 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.920903 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-home\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:53.921062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.921030 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-model-cache\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:53.921131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.921096 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvpl\" (UniqueName: \"kubernetes.io/projected/4d954ed3-9585-4249-8c87-32ee474126a7-kube-api-access-8vvpl\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:53.921205 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.921187 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-dshm\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:53.921320 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:53.921229 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-tmp-dir\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.022610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.022566 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d954ed3-9585-4249-8c87-32ee474126a7-tls-certs\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.022790 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.022695 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-home\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.022790 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.022758 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-model-cache\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.022934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.022795 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvpl\" (UniqueName: \"kubernetes.io/projected/4d954ed3-9585-4249-8c87-32ee474126a7-kube-api-access-8vvpl\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.022934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.022843 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-dshm\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.022934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.022888 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-tmp-dir\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.023305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.023277 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-model-cache\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.023395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.023309 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-tmp-dir\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.023593 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.023565 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-home\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.025381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.025348 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-dshm\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.025976 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.025956 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d954ed3-9585-4249-8c87-32ee474126a7-tls-certs\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.036173 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.036148 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvpl\" (UniqueName: \"kubernetes.io/projected/4d954ed3-9585-4249-8c87-32ee474126a7-kube-api-access-8vvpl\") pod \"test-model-a2bec7-kserve-6f8d4bb747-jd27v\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.166417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.166379 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:54.542107 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:19:54.542076 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d954ed3_9585_4249_8c87_32ee474126a7.slice/crio-385dfbc22402a720c5713bcce74209846908b47b92bd03ac3a15eb94554735c6 WatchSource:0}: Error finding container 385dfbc22402a720c5713bcce74209846908b47b92bd03ac3a15eb94554735c6: Status 404 returned error can't find the container with id 385dfbc22402a720c5713bcce74209846908b47b92bd03ac3a15eb94554735c6 Oct 31 03:19:54.542245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.542155 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v"] Oct 31 03:19:54.870851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.870769 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk"] Oct 31 03:19:54.879072 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.877254 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:19:54.887062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.884352 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b79c13\"" Oct 31 03:19:54.887062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.884352 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b79c13-dockercfg-98wk4\"" Oct 31 03:19:54.887062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.884707 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk"] Oct 31 03:19:54.906990 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.906955 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8"] Oct 31 03:19:54.912425 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.912255 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:19:54.921149 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.921119 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8"] Oct 31 03:19:54.945248 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.945218 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdt9\" (UniqueName: \"kubernetes.io/projected/8f8bee44-c20a-426c-961e-1b369f5046cb-kube-api-access-bkdt9\") pod \"payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk\" (UID: \"8f8bee44-c20a-426c-961e-1b369f5046cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:19:54.945404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.945256 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd297f3-62d3-4fc8-81ad-7973c85a402e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8\" (UID: \"9fd297f3-62d3-4fc8-81ad-7973c85a402e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:19:54.945461 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.945441 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8bee44-c20a-426c-961e-1b369f5046cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk\" (UID: \"8f8bee44-c20a-426c-961e-1b369f5046cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:19:54.945564 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:54.945541 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czbs\" (UniqueName: \"kubernetes.io/projected/9fd297f3-62d3-4fc8-81ad-7973c85a402e-kube-api-access-4czbs\") pod \"payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8\" (UID: \"9fd297f3-62d3-4fc8-81ad-7973c85a402e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:19:55.055058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.054072 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdt9\" (UniqueName: \"kubernetes.io/projected/8f8bee44-c20a-426c-961e-1b369f5046cb-kube-api-access-bkdt9\") pod \"payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk\" (UID: \"8f8bee44-c20a-426c-961e-1b369f5046cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:19:55.055058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.054154 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd297f3-62d3-4fc8-81ad-7973c85a402e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8\" (UID: \"9fd297f3-62d3-4fc8-81ad-7973c85a402e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:19:55.055058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.054280 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8bee44-c20a-426c-961e-1b369f5046cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk\" (UID: \"8f8bee44-c20a-426c-961e-1b369f5046cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:19:55.055058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.054346 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czbs\" (UniqueName: \"kubernetes.io/projected/9fd297f3-62d3-4fc8-81ad-7973c85a402e-kube-api-access-4czbs\") pod \"payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8\" (UID: \"9fd297f3-62d3-4fc8-81ad-7973c85a402e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:19:55.068058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.055966 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd297f3-62d3-4fc8-81ad-7973c85a402e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8\" (UID: \"9fd297f3-62d3-4fc8-81ad-7973c85a402e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:19:55.068058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.056716 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8bee44-c20a-426c-961e-1b369f5046cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk\" (UID: \"8f8bee44-c20a-426c-961e-1b369f5046cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:19:55.077063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.075342 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czbs\" (UniqueName: \"kubernetes.io/projected/9fd297f3-62d3-4fc8-81ad-7973c85a402e-kube-api-access-4czbs\") pod \"payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8\" (UID: \"9fd297f3-62d3-4fc8-81ad-7973c85a402e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:19:55.077063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.075973 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdt9\" (UniqueName: \"kubernetes.io/projected/8f8bee44-c20a-426c-961e-1b369f5046cb-kube-api-access-bkdt9\") pod \"payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk\" (UID: \"8f8bee44-c20a-426c-961e-1b369f5046cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:19:55.208364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.208323 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:19:55.238287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.238205 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:19:55.504093 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.503138 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" event={"ID":"4d954ed3-9585-4249-8c87-32ee474126a7","Type":"ContainerStarted","Data":"e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4"} Oct 31 03:19:55.504093 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.503194 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" event={"ID":"4d954ed3-9585-4249-8c87-32ee474126a7","Type":"ContainerStarted","Data":"385dfbc22402a720c5713bcce74209846908b47b92bd03ac3a15eb94554735c6"} Oct 31 03:19:55.545970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.542702 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" podStartSLOduration=2.542680986 podStartE2EDuration="2.542680986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:55.536795517 +0000 UTC m=+1477.217321645" watchObservedRunningTime="2026-10-31 03:19:55.542680986 +0000 UTC m=+1477.223207116" Oct 31 03:19:55.619966 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:19:55.619931 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8bee44_c20a_426c_961e_1b369f5046cb.slice/crio-db9fddd7a60e89d5d92a7db5dce2f714554c60be71d60378f680496f3419d3da WatchSource:0}: Error finding container db9fddd7a60e89d5d92a7db5dce2f714554c60be71d60378f680496f3419d3da: Status 404 returned error can't find the container with id db9fddd7a60e89d5d92a7db5dce2f714554c60be71d60378f680496f3419d3da Oct 31 03:19:55.621556 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.621526 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk"] Oct 31 03:19:55.711094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:55.704120 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8"] Oct 31 03:19:56.168087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:56.168024 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:19:56.512960 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:56.512869 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" event={"ID":"8f8bee44-c20a-426c-961e-1b369f5046cb","Type":"ContainerStarted","Data":"ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575"} Oct 31 03:19:56.512960 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:56.512934 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" event={"ID":"8f8bee44-c20a-426c-961e-1b369f5046cb","Type":"ContainerStarted","Data":"db9fddd7a60e89d5d92a7db5dce2f714554c60be71d60378f680496f3419d3da"} Oct 31 03:19:56.513209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:56.512964 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:19:56.515390 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:56.515362 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" event={"ID":"9fd297f3-62d3-4fc8-81ad-7973c85a402e","Type":"ContainerStarted","Data":"dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0"} Oct 31 03:19:56.515523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:56.515411 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" event={"ID":"9fd297f3-62d3-4fc8-81ad-7973c85a402e","Type":"ContainerStarted","Data":"875aac3072b8ecc2f466d63a407b967078459166ce6a2d876c6492cbd916e9e0"} Oct 31 03:19:56.515523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:56.515443 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:19:56.540195 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:56.540152 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" podStartSLOduration=2.540138845 podStartE2EDuration="2.540138845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:56.535494621 +0000 UTC m=+1478.216020753" watchObservedRunningTime="2026-10-31 03:19:56.540138845 +0000 UTC m=+1478.220664972" Oct 31 03:19:56.558269 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:19:56.558229 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" podStartSLOduration=2.55821599 podStartE2EDuration="2.55821599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:56.556318049 +0000 UTC m=+1478.236844178" watchObservedRunningTime="2026-10-31 03:19:56.55821599 +0000 UTC m=+1478.238742117" Oct 31 03:20:01.036195 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.036110 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s"] Oct 31 03:20:01.036754 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.036382 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" podUID="7666a98d-9d6a-439f-a141-771378ad2861" containerName="maas-api" containerID="cri-o://72e84b79749e1fb1ebc25a34eb25bfbff0b1b71cad3fcbc42c42dfac746a6896" gracePeriod=30 Oct 31 03:20:01.650105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.646089 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp"] Oct 31 03:20:01.660108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.654388 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-edb9e6-f5dfp"] Oct 31 03:20:01.673092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.673056 4456 generic.go:356] "Generic (PLEG): container finished" podID="7666a98d-9d6a-439f-a141-771378ad2861" containerID="72e84b79749e1fb1ebc25a34eb25bfbff0b1b71cad3fcbc42c42dfac746a6896" exitCode=0 Oct 31 03:20:01.673251 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.673170 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" event={"ID":"7666a98d-9d6a-439f-a141-771378ad2861","Type":"ContainerDied","Data":"72e84b79749e1fb1ebc25a34eb25bfbff0b1b71cad3fcbc42c42dfac746a6896"} Oct 31 03:20:01.728865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.723296 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:20:01.833141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.831298 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7666a98d-9d6a-439f-a141-771378ad2861-kube-api-access-qf4rt\" (UniqueName: \"kubernetes.io/projected/7666a98d-9d6a-439f-a141-771378ad2861-kube-api-access-qf4rt\") pod \"7666a98d-9d6a-439f-a141-771378ad2861\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " Oct 31 03:20:01.833141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.831404 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-metrics-certs\") pod \"7666a98d-9d6a-439f-a141-771378ad2861\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " Oct 31 03:20:01.833141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.831487 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7666a98d-9d6a-439f-a141-771378ad2861-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7666a98d-9d6a-439f-a141-771378ad2861-openshift-service-ca\") pod \"7666a98d-9d6a-439f-a141-771378ad2861\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " Oct 31 03:20:01.833141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.831687 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-maas-api-tls\") pod \"7666a98d-9d6a-439f-a141-771378ad2861\" (UID: \"7666a98d-9d6a-439f-a141-771378ad2861\") " Oct 31 03:20:01.840054 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.834835 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7666a98d-9d6a-439f-a141-771378ad2861-openshift-service-ca" pod "7666a98d-9d6a-439f-a141-771378ad2861" (UID: "7666a98d-9d6a-439f-a141-771378ad2861"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:01.847131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.843874 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-metrics-certs" pod "7666a98d-9d6a-439f-a141-771378ad2861" (UID: "7666a98d-9d6a-439f-a141-771378ad2861"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:01.847131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.844378 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7666a98d-9d6a-439f-a141-771378ad2861-kube-api-access-qf4rt" pod "7666a98d-9d6a-439f-a141-771378ad2861" (UID: "7666a98d-9d6a-439f-a141-771378ad2861"). InnerVolumeSpecName "kube-api-access-qf4rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:01.878364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.878262 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-maas-api-tls" pod "7666a98d-9d6a-439f-a141-771378ad2861" (UID: "7666a98d-9d6a-439f-a141-771378ad2861"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:01.933934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.933896 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-maas-api-tls\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:01.934154 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.933955 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf4rt\" (UniqueName: \"kubernetes.io/projected/7666a98d-9d6a-439f-a141-771378ad2861-kube-api-access-qf4rt\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:01.934154 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.933968 4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7666a98d-9d6a-439f-a141-771378ad2861-metrics-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:01.934154 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:01.933977 4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7666a98d-9d6a-439f-a141-771378ad2861-openshift-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:02.137566 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:20:02.137514 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:20:02.437276 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:02.437238 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e38aebf-2b42-486d-8650-f9ca54007d3e" path="/var/lib/kubelet/pods/8e38aebf-2b42-486d-8650-f9ca54007d3e/volumes" Oct 31 03:20:02.692110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:02.691795 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" event={"ID":"7666a98d-9d6a-439f-a141-771378ad2861","Type":"ContainerDied","Data":"600f21c48e85ede2d17f52fe2350762f00f8b2f6cdb07af1e36a4524da9f0976"} Oct 31 03:20:02.692110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:02.691845 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s" Oct 31 03:20:02.692110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:02.691882 4456 scope.go:122] "RemoveContainer" containerID="72e84b79749e1fb1ebc25a34eb25bfbff0b1b71cad3fcbc42c42dfac746a6896" Oct 31 03:20:02.732774 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:02.732738 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s"] Oct 31 03:20:02.737580 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:02.737553 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-edb9e6-9944fbdc8-fcs7s"] Oct 31 03:20:04.168621 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.168581 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:20:04.182451 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.182413 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:20:04.439349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.439261 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7666a98d-9d6a-439f-a141-771378ad2861" path="/var/lib/kubelet/pods/7666a98d-9d6a-439f-a141-771378ad2861/volumes" Oct 31 03:20:04.455423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.455386 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx"] Oct 31 03:20:04.457175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.456597 4456 cpu_manager.go:414] "containerMap: removing container" podUID="7666a98d-9d6a-439f-a141-771378ad2861" containerName="maas-api" Oct 31 03:20:04.457175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.456619 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7666a98d-9d6a-439f-a141-771378ad2861" containerName="maas-api" Oct 31 03:20:04.457175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.456784 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7666a98d-9d6a-439f-a141-771378ad2861" containerName="maas-api" Oct 31 03:20:04.472082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.465736 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.472082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.467644 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx"] Oct 31 03:20:04.481303 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.481272 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7269ae\"/\"test-model-7269ae-kserve-self-signed-certs\"" Oct 31 03:20:04.481454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.481365 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7269ae\"/\"openshift-service-ca.crt\"" Oct 31 03:20:04.481454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.481426 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7269ae\"/\"kube-root-ca.crt\"" Oct 31 03:20:04.481560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.481472 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7269ae\"/\"default-dockercfg-qq2lp\"" Oct 31 03:20:04.585529 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.585489 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-dshm\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.585750 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.585632 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-tmp-dir\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.585750 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.585701 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-home\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.585878 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.585809 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-model-cache\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.585878 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.585850 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8a0ce3-d044-4698-9ec5-585412658ec1-tls-certs\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.586021 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.586003 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttll9\" (UniqueName: \"kubernetes.io/projected/da8a0ce3-d044-4698-9ec5-585412658ec1-kube-api-access-ttll9\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.700127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.690986 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-tmp-dir\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.700127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.691082 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-home\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.700127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.691121 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-model-cache\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.700127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.691149 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8a0ce3-d044-4698-9ec5-585412658ec1-tls-certs\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.700127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.691298 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttll9\" (UniqueName: \"kubernetes.io/projected/da8a0ce3-d044-4698-9ec5-585412658ec1-kube-api-access-ttll9\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.700127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.691440 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-dshm\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.700127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.692296 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-model-cache\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.700127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.692572 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-tmp-dir\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.700127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.692826 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-home\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.700127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.700062 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8a0ce3-d044-4698-9ec5-585412658ec1-tls-certs\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.754713 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.754655 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttll9\" (UniqueName: \"kubernetes.io/projected/da8a0ce3-d044-4698-9ec5-585412658ec1-kube-api-access-ttll9\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.778309 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.777794 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-dshm\") pod \"test-model-7269ae-kserve-6d4bfcdc5-bthlx\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.791666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.791314 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:04.897819 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:04.897710 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:20:05.196893 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:05.196833 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx"] Oct 31 03:20:05.415135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:05.409074 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4"] Oct 31 03:20:05.416856 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:05.416826 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fd580-8m6s4"] Oct 31 03:20:05.839064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:05.829136 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" event={"ID":"da8a0ce3-d044-4698-9ec5-585412658ec1","Type":"ContainerStarted","Data":"7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040"} Oct 31 03:20:05.839064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:05.829185 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" event={"ID":"da8a0ce3-d044-4698-9ec5-585412658ec1","Type":"ContainerStarted","Data":"67b2f6516c110f247fd0abaa19b86d8898ede4d139a6a8e931327f7ffd8dbf32"} Oct 31 03:20:05.885375 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:05.885310 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" podStartSLOduration=1.88528788 podStartE2EDuration="1.88528788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:05.873236593 +0000 UTC m=+1487.553762720" watchObservedRunningTime="2026-10-31 03:20:05.88528788 +0000 UTC m=+1487.565814008" Oct 31 03:20:06.437180 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:06.437148 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff69a1c7-9b7e-4486-ab3f-1784b34e0e58" path="/var/lib/kubelet/pods/ff69a1c7-9b7e-4486-ab3f-1784b34e0e58/volumes" Oct 31 03:20:06.795395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:06.795307 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:07.530089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:07.528921 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:20:07.530564 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:07.530253 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:20:10.320685 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:10.320648 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs"] Oct 31 03:20:10.321520 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:10.321483 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" containerID="cri-o://d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e" gracePeriod=30 Oct 31 03:20:10.372392 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:10.372355 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97"] Oct 31 03:20:10.375730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:10.375695 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" containerID="cri-o://6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188" gracePeriod=30 Oct 31 03:20:10.377318 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:10.377297 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9"] Oct 31 03:20:10.377677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:10.377639 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" containerID="cri-o://b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1" gracePeriod=30 Oct 31 03:20:14.837063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:14.836461 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:14.915816 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:14.915768 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:15.207206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.206850 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:20:15.606438 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.605323 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:20:15.661061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.659236 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-kube-api-access-zhz8q\" (UniqueName: \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-kube-api-access-zhz8q\") pod \"4f59a4f5-d136-4c9b-a557-792bb6b71657\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " Oct 31 03:20:15.661061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.659396 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-credential-socket\") pod \"4f59a4f5-d136-4c9b-a557-792bb6b71657\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " Oct 31 03:20:15.661061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.659544 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f59a4f5-d136-4c9b-a557-792bb6b71657-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f59a4f5-d136-4c9b-a557-792bb6b71657-istiod-ca-cert\") pod \"4f59a4f5-d136-4c9b-a557-792bb6b71657\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " Oct 31 03:20:15.661061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.659604 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-token\" (UniqueName: \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-token\") pod \"4f59a4f5-d136-4c9b-a557-792bb6b71657\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " Oct 31 03:20:15.661061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.659802 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-envoy\") pod \"4f59a4f5-d136-4c9b-a557-792bb6b71657\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " Oct 31 03:20:15.661061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.659953 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-data\") pod \"4f59a4f5-d136-4c9b-a557-792bb6b71657\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " Oct 31 03:20:15.661061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.660346 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-certs\") pod \"4f59a4f5-d136-4c9b-a557-792bb6b71657\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " Oct 31 03:20:15.661061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.660661 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-socket\") pod \"4f59a4f5-d136-4c9b-a557-792bb6b71657\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " Oct 31 03:20:15.661061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.660750 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-podinfo\") pod \"4f59a4f5-d136-4c9b-a557-792bb6b71657\" (UID: \"4f59a4f5-d136-4c9b-a557-792bb6b71657\") " Oct 31 03:20:15.677815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.667740 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-credential-socket" pod "4f59a4f5-d136-4c9b-a557-792bb6b71657" (UID: "4f59a4f5-d136-4c9b-a557-792bb6b71657"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.677815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.670653 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f59a4f5-d136-4c9b-a557-792bb6b71657-istiod-ca-cert" pod "4f59a4f5-d136-4c9b-a557-792bb6b71657" (UID: "4f59a4f5-d136-4c9b-a557-792bb6b71657"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:15.677815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.671287 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-data" pod "4f59a4f5-d136-4c9b-a557-792bb6b71657" (UID: "4f59a4f5-d136-4c9b-a557-792bb6b71657"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.680363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.680016 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-certs" pod "4f59a4f5-d136-4c9b-a557-792bb6b71657" (UID: "4f59a4f5-d136-4c9b-a557-792bb6b71657"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.690250 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.689485 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.690250 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.689581 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.690250 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.689600 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.690250 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.689615 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f59a4f5-d136-4c9b-a557-792bb6b71657-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.690250 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.689901 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-socket" pod "4f59a4f5-d136-4c9b-a557-792bb6b71657" (UID: "4f59a4f5-d136-4c9b-a557-792bb6b71657"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.709932 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.709859 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-podinfo" pod "4f59a4f5-d136-4c9b-a557-792bb6b71657" (UID: "4f59a4f5-d136-4c9b-a557-792bb6b71657"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:20:15.710238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.710191 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-kube-api-access-zhz8q" pod "4f59a4f5-d136-4c9b-a557-792bb6b71657" (UID: "4f59a4f5-d136-4c9b-a557-792bb6b71657"). InnerVolumeSpecName "kube-api-access-zhz8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:15.710708 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.710678 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-token" pod "4f59a4f5-d136-4c9b-a557-792bb6b71657" (UID: "4f59a4f5-d136-4c9b-a557-792bb6b71657"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:15.715099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.715030 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-envoy" pod "4f59a4f5-d136-4c9b-a557-792bb6b71657" (UID: "4f59a4f5-d136-4c9b-a557-792bb6b71657"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.729959 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.729925 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:20:15.792106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.791930 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.792106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.791958 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.792106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.791969 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f59a4f5-d136-4c9b-a557-792bb6b71657-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.792106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.791977 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f59a4f5-d136-4c9b-a557-792bb6b71657-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.792106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.791988 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhz8q\" (UniqueName: \"kubernetes.io/projected/4f59a4f5-d136-4c9b-a557-792bb6b71657-kube-api-access-zhz8q\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.801663 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.801633 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:20:15.892735 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.892705 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-podinfo\") pod \"c1605d8d-256f-42f5-b3fc-448e3dde1950\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " Oct 31 03:20:15.893186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.892777 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-envoy\") pod \"c1605d8d-256f-42f5-b3fc-448e3dde1950\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " Oct 31 03:20:15.893186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.892798 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b501950-8aba-4108-9320-d59694798c0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b501950-8aba-4108-9320-d59694798c0a-istiod-ca-cert\") pod \"7b501950-8aba-4108-9320-d59694798c0a\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " Oct 31 03:20:15.893186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.892906 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-credential-socket\") pod \"7b501950-8aba-4108-9320-d59694798c0a\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " Oct 31 03:20:15.893186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.892953 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-token\" (UniqueName: \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-token\") pod \"c1605d8d-256f-42f5-b3fc-448e3dde1950\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " Oct 31 03:20:15.893186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.892993 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-certs\") pod \"7b501950-8aba-4108-9320-d59694798c0a\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " Oct 31 03:20:15.893186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893047 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-istio-token\" (UniqueName: \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-istio-token\") pod \"7b501950-8aba-4108-9320-d59694798c0a\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " Oct 31 03:20:15.893186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893107 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-kube-api-access-plhvx\" (UniqueName: \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-kube-api-access-plhvx\") pod \"7b501950-8aba-4108-9320-d59694798c0a\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " Oct 31 03:20:15.893186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893167 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-credential-socket\") pod \"c1605d8d-256f-42f5-b3fc-448e3dde1950\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893225 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-credential-socket" pod "7b501950-8aba-4108-9320-d59694798c0a" (UID: "7b501950-8aba-4108-9320-d59694798c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893250 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b501950-8aba-4108-9320-d59694798c0a-istiod-ca-cert" pod "7b501950-8aba-4108-9320-d59694798c0a" (UID: "7b501950-8aba-4108-9320-d59694798c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893307 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-socket\") pod \"7b501950-8aba-4108-9320-d59694798c0a\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893366 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-data\") pod \"7b501950-8aba-4108-9320-d59694798c0a\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893364 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-certs" pod "7b501950-8aba-4108-9320-d59694798c0a" (UID: "7b501950-8aba-4108-9320-d59694798c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893410 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1605d8d-256f-42f5-b3fc-448e3dde1950-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1605d8d-256f-42f5-b3fc-448e3dde1950-istiod-ca-cert\") pod \"c1605d8d-256f-42f5-b3fc-448e3dde1950\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893464 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-certs\") pod \"c1605d8d-256f-42f5-b3fc-448e3dde1950\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893529 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-envoy\") pod \"7b501950-8aba-4108-9320-d59694798c0a\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893567 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-kube-api-access-tj8fd\" (UniqueName: \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-kube-api-access-tj8fd\") pod \"c1605d8d-256f-42f5-b3fc-448e3dde1950\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893592 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-credential-socket" pod "c1605d8d-256f-42f5-b3fc-448e3dde1950" (UID: "c1605d8d-256f-42f5-b3fc-448e3dde1950"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893602 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-socket\") pod \"c1605d8d-256f-42f5-b3fc-448e3dde1950\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " Oct 31 03:20:15.893645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893593 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-socket" pod "7b501950-8aba-4108-9320-d59694798c0a" (UID: "7b501950-8aba-4108-9320-d59694798c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.894284 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893657 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b501950-8aba-4108-9320-d59694798c0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b501950-8aba-4108-9320-d59694798c0a-istio-podinfo\") pod \"7b501950-8aba-4108-9320-d59694798c0a\" (UID: \"7b501950-8aba-4108-9320-d59694798c0a\") " Oct 31 03:20:15.894284 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893694 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-data\") pod \"c1605d8d-256f-42f5-b3fc-448e3dde1950\" (UID: \"c1605d8d-256f-42f5-b3fc-448e3dde1950\") " Oct 31 03:20:15.894284 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893738 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-data" pod "7b501950-8aba-4108-9320-d59694798c0a" (UID: "7b501950-8aba-4108-9320-d59694798c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.894284 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893800 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-certs" pod "c1605d8d-256f-42f5-b3fc-448e3dde1950" (UID: "c1605d8d-256f-42f5-b3fc-448e3dde1950"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.894284 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.893955 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-socket" pod "c1605d8d-256f-42f5-b3fc-448e3dde1950" (UID: "c1605d8d-256f-42f5-b3fc-448e3dde1950"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.894284 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894146 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1605d8d-256f-42f5-b3fc-448e3dde1950-istiod-ca-cert" pod "c1605d8d-256f-42f5-b3fc-448e3dde1950" (UID: "c1605d8d-256f-42f5-b3fc-448e3dde1950"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:15.894284 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894232 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-data" pod "c1605d8d-256f-42f5-b3fc-448e3dde1950" (UID: "c1605d8d-256f-42f5-b3fc-448e3dde1950"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.894611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894548 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.894611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894573 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.894611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894589 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.894611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894603 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.894794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894616 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1605d8d-256f-42f5-b3fc-448e3dde1950-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.894794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894631 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.894794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894645 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.894794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894659 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.894794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894672 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b501950-8aba-4108-9320-d59694798c0a-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.894794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.894685 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.895787 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.895507 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-envoy" pod "c1605d8d-256f-42f5-b3fc-448e3dde1950" (UID: "c1605d8d-256f-42f5-b3fc-448e3dde1950"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.895787 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.895753 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-kube-api-access-plhvx" pod "7b501950-8aba-4108-9320-d59694798c0a" (UID: "7b501950-8aba-4108-9320-d59694798c0a"). InnerVolumeSpecName "kube-api-access-plhvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:15.896329 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.896308 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-token" pod "c1605d8d-256f-42f5-b3fc-448e3dde1950" (UID: "c1605d8d-256f-42f5-b3fc-448e3dde1950"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:15.896489 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.896470 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-kube-api-access-tj8fd" pod "c1605d8d-256f-42f5-b3fc-448e3dde1950" (UID: "c1605d8d-256f-42f5-b3fc-448e3dde1950"). InnerVolumeSpecName "kube-api-access-tj8fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:15.896563 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.896500 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-podinfo" pod "c1605d8d-256f-42f5-b3fc-448e3dde1950" (UID: "c1605d8d-256f-42f5-b3fc-448e3dde1950"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:20:15.896710 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.896694 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b501950-8aba-4108-9320-d59694798c0a-istio-podinfo" pod "7b501950-8aba-4108-9320-d59694798c0a" (UID: "7b501950-8aba-4108-9320-d59694798c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:20:15.896854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.896827 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-istio-token" pod "7b501950-8aba-4108-9320-d59694798c0a" (UID: "7b501950-8aba-4108-9320-d59694798c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:15.897285 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.897268 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-envoy" pod "7b501950-8aba-4108-9320-d59694798c0a" (UID: "7b501950-8aba-4108-9320-d59694798c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:15.995789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.995710 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plhvx\" (UniqueName: \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-kube-api-access-plhvx\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.995789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.995744 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b501950-8aba-4108-9320-d59694798c0a-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.995789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.995781 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj8fd\" (UniqueName: \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-kube-api-access-tj8fd\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.995789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.995791 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b501950-8aba-4108-9320-d59694798c0a-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.996031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.995800 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.996031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.995809 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.996031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.995819 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1605d8d-256f-42f5-b3fc-448e3dde1950-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:15.996031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:15.995828 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b501950-8aba-4108-9320-d59694798c0a-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:16.204182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.204142 4456 generic.go:356] "Generic (PLEG): container finished" podID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerID="6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188" exitCode=0 Oct 31 03:20:16.204349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.204214 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" Oct 31 03:20:16.204349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.204226 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" event={"ID":"c1605d8d-256f-42f5-b3fc-448e3dde1950","Type":"ContainerDied","Data":"6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188"} Oct 31 03:20:16.204349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.204314 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97" event={"ID":"c1605d8d-256f-42f5-b3fc-448e3dde1950","Type":"ContainerDied","Data":"aa5aaf65280fcb2b4e1143267b30b11c020fca9a6dfc7956c30ae0a57c2eff16"} Oct 31 03:20:16.204349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.204339 4456 scope.go:122] "RemoveContainer" containerID="6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188" Oct 31 03:20:16.206906 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.206879 4456 generic.go:356] "Generic (PLEG): container finished" podID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerID="d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e" exitCode=0 Oct 31 03:20:16.209562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.206927 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" event={"ID":"4f59a4f5-d136-4c9b-a557-792bb6b71657","Type":"ContainerDied","Data":"d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e"} Oct 31 03:20:16.209562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.206990 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" event={"ID":"4f59a4f5-d136-4c9b-a557-792bb6b71657","Type":"ContainerDied","Data":"63d8c35befae4b44ebab085ded9b8db8eb334fddcda2c66aec27df941320e83d"} Oct 31 03:20:16.209562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.207135 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs" Oct 31 03:20:16.212931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.212910 4456 generic.go:356] "Generic (PLEG): container finished" podID="7b501950-8aba-4108-9320-d59694798c0a" containerID="b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1" exitCode=0 Oct 31 03:20:16.213771 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.213753 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" Oct 31 03:20:16.215237 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.214748 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" event={"ID":"7b501950-8aba-4108-9320-d59694798c0a","Type":"ContainerDied","Data":"b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1"} Oct 31 03:20:16.215237 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.214810 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9" event={"ID":"7b501950-8aba-4108-9320-d59694798c0a","Type":"ContainerDied","Data":"5fa2e057b2a06589b2615e5d8635e02e0b46b9598783486c03b16ef837c8c53c"} Oct 31 03:20:16.218490 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.218453 4456 scope.go:122] "RemoveContainer" containerID="6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188" Oct 31 03:20:16.218737 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:20:16.218713 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188\": container with ID starting with 6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188 not found: ID does not exist" containerID="6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188" Oct 31 03:20:16.218821 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.218747 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188"} err="failed to get container status \"6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188\": rpc error: code = NotFound desc = could not find container \"6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188\": container with ID starting with 6b6de25f66accd3ee7f755921356cf69d50d854d45f608d0c7b22bc003346188 not found: ID does not exist" Oct 31 03:20:16.218821 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.218768 4456 scope.go:122] "RemoveContainer" containerID="d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e" Oct 31 03:20:16.232282 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.232262 4456 scope.go:122] "RemoveContainer" containerID="d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e" Oct 31 03:20:16.232571 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:20:16.232539 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e\": container with ID starting with d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e not found: ID does not exist" containerID="d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e" Oct 31 03:20:16.232661 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.232579 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e"} err="failed to get container status \"d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e\": rpc error: code = NotFound desc = could not find container \"d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e\": container with ID starting with d0371d7a2298360f84bea049f3d33b71b8205a20e8534acd2471ca5908cbd58e not found: ID does not exist" Oct 31 03:20:16.232661 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.232601 4456 scope.go:122] "RemoveContainer" containerID="b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1" Oct 31 03:20:16.243670 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.243653 4456 scope.go:122] "RemoveContainer" containerID="b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1" Oct 31 03:20:16.243955 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:20:16.243935 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1\": container with ID starting with b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1 not found: ID does not exist" containerID="b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1" Oct 31 03:20:16.244006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.243962 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1"} err="failed to get container status \"b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1\": rpc error: code = NotFound desc = could not find container \"b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1\": container with ID starting with b3492675c10f895c8d1f4aec767ecfbcfd6e9b6aed12b944c7d6084db5fd6af1 not found: ID does not exist" Oct 31 03:20:16.252315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.252249 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97"] Oct 31 03:20:16.270092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.268365 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7sf97"] Oct 31 03:20:16.309427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.309396 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9"] Oct 31 03:20:16.328453 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.328422 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-7chm9"] Oct 31 03:20:16.347387 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.347353 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs"] Oct 31 03:20:16.355313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.355284 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-g5vbs"] Oct 31 03:20:16.440220 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.440187 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" path="/var/lib/kubelet/pods/4f59a4f5-d136-4c9b-a557-792bb6b71657/volumes" Oct 31 03:20:16.440794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.440758 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b501950-8aba-4108-9320-d59694798c0a" path="/var/lib/kubelet/pods/7b501950-8aba-4108-9320-d59694798c0a/volumes" Oct 31 03:20:16.443090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:16.441413 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" path="/var/lib/kubelet/pods/c1605d8d-256f-42f5-b3fc-448e3dde1950/volumes" Oct 31 03:20:18.556355 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.556322 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj"] Oct 31 03:20:18.557025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.557009 4456 cpu_manager.go:414] "containerMap: removing container" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" Oct 31 03:20:18.557085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.557027 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" Oct 31 03:20:18.557085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.557060 4456 cpu_manager.go:414] "containerMap: removing container" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" Oct 31 03:20:18.557085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.557068 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" Oct 31 03:20:18.557185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.557124 4456 cpu_manager.go:414] "containerMap: removing container" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" Oct 31 03:20:18.557185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.557130 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" Oct 31 03:20:18.557256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.557246 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b501950-8aba-4108-9320-d59694798c0a" containerName="istio-proxy" Oct 31 03:20:18.557291 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.557262 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f59a4f5-d136-4c9b-a557-792bb6b71657" containerName="istio-proxy" Oct 31 03:20:18.557291 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.557271 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1605d8d-256f-42f5-b3fc-448e3dde1950" containerName="istio-proxy" Oct 31 03:20:18.562625 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.562605 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:18.573882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.573854 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj"] Oct 31 03:20:18.593978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.593940 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b8712-blwnj\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:18.594176 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.594005 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b8712-blwnj\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:18.594402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.594382 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gpcf\" (UniqueName: \"kubernetes.io/projected/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-kube-api-access-6gpcf\") pod \"maas-api-revoke-keys-e2e-api-b-3b8712-blwnj\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:18.668047 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.667975 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:20:18.668209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.668071 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:20:18.695774 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.695734 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gpcf\" (UniqueName: \"kubernetes.io/projected/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-kube-api-access-6gpcf\") pod \"maas-api-revoke-keys-e2e-api-b-3b8712-blwnj\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:18.695937 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.695853 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b8712-blwnj\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:18.695937 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.695891 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b8712-blwnj\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:18.696813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.696787 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b8712-blwnj\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:18.698741 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.698707 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b8712-blwnj\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:18.709472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.709449 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gpcf\" (UniqueName: \"kubernetes.io/projected/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-kube-api-access-6gpcf\") pod \"maas-api-revoke-keys-e2e-api-b-3b8712-blwnj\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:18.896176 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:18.896087 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:19.092655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:19.092614 4456 scope.go:122] "RemoveContainer" containerID="66f73d4c54771be7b70924060ef637dce1a3996cb24c001fc31f0f974360fa33" Oct 31 03:20:19.231740 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:20:19.231706 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fc4a13_d8f5_4488_a51c_8a80b8c8493b.slice/crio-2942bcd5e3ecc48c77c2c6a5bd75b37e7ed81b4cda01449901f1036ef45ac8e1 WatchSource:0}: Error finding container 2942bcd5e3ecc48c77c2c6a5bd75b37e7ed81b4cda01449901f1036ef45ac8e1: Status 404 returned error can't find the container with id 2942bcd5e3ecc48c77c2c6a5bd75b37e7ed81b4cda01449901f1036ef45ac8e1 Oct 31 03:20:19.234299 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:19.234263 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj"] Oct 31 03:20:19.305081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:19.305032 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" event={"ID":"79fc4a13-d8f5-4488-a51c-8a80b8c8493b","Type":"ContainerStarted","Data":"2942bcd5e3ecc48c77c2c6a5bd75b37e7ed81b4cda01449901f1036ef45ac8e1"} Oct 31 03:20:20.314659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:20.314617 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" event={"ID":"79fc4a13-d8f5-4488-a51c-8a80b8c8493b","Type":"ContainerStarted","Data":"3635681c4934fdccb0c70d214e6bff7f64c431f72859ce5408be48efda72cdfb"} Oct 31 03:20:20.368544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:20.368492 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" podStartSLOduration=2.368477693 podStartE2EDuration="2.368477693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:20.365406056 +0000 UTC m=+1502.045932184" watchObservedRunningTime="2026-10-31 03:20:20.368477693 +0000 UTC m=+1502.049003875" Oct 31 03:20:21.324578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:21.324524 4456 generic.go:356] "Generic (PLEG): container finished" podID="79fc4a13-d8f5-4488-a51c-8a80b8c8493b" containerID="3635681c4934fdccb0c70d214e6bff7f64c431f72859ce5408be48efda72cdfb" exitCode=0 Oct 31 03:20:21.325061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:21.324590 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" event={"ID":"79fc4a13-d8f5-4488-a51c-8a80b8c8493b","Type":"ContainerDied","Data":"3635681c4934fdccb0c70d214e6bff7f64c431f72859ce5408be48efda72cdfb"} Oct 31 03:20:21.935650 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:21.935608 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z"] Oct 31 03:20:21.940385 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:21.940361 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:20:21.954163 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:21.954134 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5"] Oct 31 03:20:21.956273 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:21.956247 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bbbc81\"" Oct 31 03:20:21.963028 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:21.962996 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:20:21.995609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:21.995559 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z"] Oct 31 03:20:22.009993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.009962 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5"] Oct 31 03:20:22.042350 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.042317 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdlr4\" (UniqueName: \"kubernetes.io/projected/2dcf78bf-69a6-42b8-99c9-ee5f51323973-kube-api-access-cdlr4\") pod \"payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5\" (UID: \"2dcf78bf-69a6-42b8-99c9-ee5f51323973\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:20:22.042509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.042373 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186c3940-33eb-42f6-93e2-def354123519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z\" (UID: \"186c3940-33eb-42f6-93e2-def354123519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:20:22.042509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.042390 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh8zs\" (UniqueName: \"kubernetes.io/projected/186c3940-33eb-42f6-93e2-def354123519-kube-api-access-dh8zs\") pod \"payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z\" (UID: \"186c3940-33eb-42f6-93e2-def354123519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:20:22.042587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.042519 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dcf78bf-69a6-42b8-99c9-ee5f51323973-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5\" (UID: \"2dcf78bf-69a6-42b8-99c9-ee5f51323973\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:20:22.144100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.144025 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdlr4\" (UniqueName: \"kubernetes.io/projected/2dcf78bf-69a6-42b8-99c9-ee5f51323973-kube-api-access-cdlr4\") pod \"payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5\" (UID: \"2dcf78bf-69a6-42b8-99c9-ee5f51323973\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:20:22.144307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.144135 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186c3940-33eb-42f6-93e2-def354123519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z\" (UID: \"186c3940-33eb-42f6-93e2-def354123519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:20:22.144307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.144158 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8zs\" (UniqueName: \"kubernetes.io/projected/186c3940-33eb-42f6-93e2-def354123519-kube-api-access-dh8zs\") pod \"payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z\" (UID: \"186c3940-33eb-42f6-93e2-def354123519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:20:22.144307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.144226 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dcf78bf-69a6-42b8-99c9-ee5f51323973-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5\" (UID: \"2dcf78bf-69a6-42b8-99c9-ee5f51323973\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:20:22.145131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.145029 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dcf78bf-69a6-42b8-99c9-ee5f51323973-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5\" (UID: \"2dcf78bf-69a6-42b8-99c9-ee5f51323973\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:20:22.145887 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.145867 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186c3940-33eb-42f6-93e2-def354123519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z\" (UID: \"186c3940-33eb-42f6-93e2-def354123519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:20:22.166546 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.166006 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdlr4\" (UniqueName: \"kubernetes.io/projected/2dcf78bf-69a6-42b8-99c9-ee5f51323973-kube-api-access-cdlr4\") pod \"payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5\" (UID: \"2dcf78bf-69a6-42b8-99c9-ee5f51323973\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:20:22.199135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.196764 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8zs\" (UniqueName: \"kubernetes.io/projected/186c3940-33eb-42f6-93e2-def354123519-kube-api-access-dh8zs\") pod \"payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z\" (UID: \"186c3940-33eb-42f6-93e2-def354123519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:20:22.265089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.264394 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:20:22.300753 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.300674 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:20:22.644014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.643985 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:22.661416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.660562 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-openshift-ca\") pod \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " Oct 31 03:20:22.661416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.660623 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-kube-api-access-6gpcf\" (UniqueName: \"kubernetes.io/projected/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-kube-api-access-6gpcf\") pod \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " Oct 31 03:20:22.661416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.660733 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-xks-ca\") pod \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\" (UID: \"79fc4a13-d8f5-4488-a51c-8a80b8c8493b\") " Oct 31 03:20:22.662246 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:20:22.661846 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcf78bf_69a6_42b8_99c9_ee5f51323973.slice/crio-5bd5a87789ea8eb94e1caea226e30792a2303898241a4e83a255651a47a7f36a WatchSource:0}: Error finding container 5bd5a87789ea8eb94e1caea226e30792a2303898241a4e83a255651a47a7f36a: Status 404 returned error can't find the container with id 5bd5a87789ea8eb94e1caea226e30792a2303898241a4e83a255651a47a7f36a Oct 31 03:20:22.662246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.662135 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-openshift-ca" pod "79fc4a13-d8f5-4488-a51c-8a80b8c8493b" (UID: "79fc4a13-d8f5-4488-a51c-8a80b8c8493b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:22.663972 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.663517 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-xks-ca" pod "79fc4a13-d8f5-4488-a51c-8a80b8c8493b" (UID: "79fc4a13-d8f5-4488-a51c-8a80b8c8493b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:22.668717 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.666216 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-kube-api-access-6gpcf" pod "79fc4a13-d8f5-4488-a51c-8a80b8c8493b" (UID: "79fc4a13-d8f5-4488-a51c-8a80b8c8493b"). InnerVolumeSpecName "kube-api-access-6gpcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:22.684150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.684124 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z"] Oct 31 03:20:22.691367 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.691340 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5"] Oct 31 03:20:22.763218 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.763181 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:22.763218 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.763216 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gpcf\" (UniqueName: \"kubernetes.io/projected/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-kube-api-access-6gpcf\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:22.763441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:22.763233 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fc4a13-d8f5-4488-a51c-8a80b8c8493b-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:23.355912 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.355862 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" event={"ID":"2dcf78bf-69a6-42b8-99c9-ee5f51323973","Type":"ContainerStarted","Data":"960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de"} Oct 31 03:20:23.356206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.356175 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" event={"ID":"2dcf78bf-69a6-42b8-99c9-ee5f51323973","Type":"ContainerStarted","Data":"5bd5a87789ea8eb94e1caea226e30792a2303898241a4e83a255651a47a7f36a"} Oct 31 03:20:23.356421 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.356398 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:20:23.363091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.360612 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" event={"ID":"186c3940-33eb-42f6-93e2-def354123519","Type":"ContainerStarted","Data":"dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c"} Oct 31 03:20:23.363091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.360668 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" event={"ID":"186c3940-33eb-42f6-93e2-def354123519","Type":"ContainerStarted","Data":"5a34d61b1965019a8b88499aef8a7b75f2e0eabefd1d9d5a55a9064a95f41727"} Oct 31 03:20:23.363091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.363052 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:20:23.367876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.367848 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" event={"ID":"79fc4a13-d8f5-4488-a51c-8a80b8c8493b","Type":"ContainerDied","Data":"2942bcd5e3ecc48c77c2c6a5bd75b37e7ed81b4cda01449901f1036ef45ac8e1"} Oct 31 03:20:23.368235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.368147 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj" Oct 31 03:20:23.368378 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.368360 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2942bcd5e3ecc48c77c2c6a5bd75b37e7ed81b4cda01449901f1036ef45ac8e1" Oct 31 03:20:23.392458 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.392405 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" podStartSLOduration=2.392390745 podStartE2EDuration="2.392390745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:23.390869099 +0000 UTC m=+1505.071395223" watchObservedRunningTime="2026-10-31 03:20:23.392390745 +0000 UTC m=+1505.072916873" Oct 31 03:20:23.418959 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.418907 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" podStartSLOduration=2.41889275 podStartE2EDuration="2.41889275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:23.415569269 +0000 UTC m=+1505.096095394" watchObservedRunningTime="2026-10-31 03:20:23.41889275 +0000 UTC m=+1505.099418877" Oct 31 03:20:23.606993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.592299 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx"] Oct 31 03:20:23.606993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.593628 4456 cpu_manager.go:414] "containerMap: removing container" podUID="79fc4a13-d8f5-4488-a51c-8a80b8c8493b" containerName="revoke-keys" Oct 31 03:20:23.606993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.593652 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fc4a13-d8f5-4488-a51c-8a80b8c8493b" containerName="revoke-keys" Oct 31 03:20:23.606993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.593811 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fc4a13-d8f5-4488-a51c-8a80b8c8493b" containerName="revoke-keys" Oct 31 03:20:23.606993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.605616 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.608919 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.608235 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx"] Oct 31 03:20:23.616668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.616636 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-62bf00-openshift-default-dockercfg-xf6pr\"" Oct 31 03:20:23.713289 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.713245 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.713765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.713336 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.713765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.713379 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.713765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.713431 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.713765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.713555 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjhd\" (UniqueName: \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-kube-api-access-nrjhd\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.713765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.713631 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.713765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.713738 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.714111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.713801 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.714111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.713874 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b329d07-76ed-4336-be5d-ac8fc49c0276-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.815163 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815114 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b329d07-76ed-4336-be5d-ac8fc49c0276-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.815359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815182 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.815359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815226 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.815359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815277 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.815539 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815469 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.815591 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815534 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjhd\" (UniqueName: \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-kube-api-access-nrjhd\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.815687 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815656 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.815774 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815756 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.815876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815860 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.815954 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815919 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.816008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815949 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.816008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.815974 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b329d07-76ed-4336-be5d-ac8fc49c0276-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.816379 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.816356 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.816477 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.816401 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.818129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.818106 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.818241 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.818223 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.829747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.829720 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjhd\" (UniqueName: \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-kube-api-access-nrjhd\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.829847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.829719 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-lglfx\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:23.932325 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:23.932293 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:24.237871 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:20:24.236844 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b329d07_76ed_4336_be5d_ac8fc49c0276.slice/crio-5af9d7a1bc0d126280a9aaa21b1e5c448dfb888fe47b0e3ef177eda810ecc2f2 WatchSource:0}: Error finding container 5af9d7a1bc0d126280a9aaa21b1e5c448dfb888fe47b0e3ef177eda810ecc2f2: Status 404 returned error can't find the container with id 5af9d7a1bc0d126280a9aaa21b1e5c448dfb888fe47b0e3ef177eda810ecc2f2 Oct 31 03:20:24.237871 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:24.236955 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx"] Oct 31 03:20:24.240542 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:24.240492 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:20:24.240667 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:24.240590 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:20:24.240667 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:24.240628 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:20:24.389829 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:24.389137 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" event={"ID":"5b329d07-76ed-4336-be5d-ac8fc49c0276","Type":"ContainerStarted","Data":"aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4"} Oct 31 03:20:24.389829 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:24.389230 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" event={"ID":"5b329d07-76ed-4336-be5d-ac8fc49c0276","Type":"ContainerStarted","Data":"5af9d7a1bc0d126280a9aaa21b1e5c448dfb888fe47b0e3ef177eda810ecc2f2"} Oct 31 03:20:24.417451 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:24.417392 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" podStartSLOduration=1.417376729 podStartE2EDuration="1.417376729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:24.416695805 +0000 UTC m=+1506.097221934" watchObservedRunningTime="2026-10-31 03:20:24.417376729 +0000 UTC m=+1506.097902855" Oct 31 03:20:24.932432 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:24.932393 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:25.936594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:25.936554 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:20:25.937173 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:25.937147 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:20:26.440605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.440568 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q"] Oct 31 03:20:26.493122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.477461 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.500694 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.500504 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q"] Oct 31 03:20:26.558062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.553300 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7269ae-openshift-default-dockercfg-kfww6\"" Oct 31 03:20:26.604609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.603138 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.604609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.603266 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.604609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.603335 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fzzp\" (UniqueName: \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-kube-api-access-7fzzp\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.604609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.603620 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.604609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.603696 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.604609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.604210 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.604609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.604441 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.604609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.604472 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.605208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.604644 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.706090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.705976 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.706331 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.706154 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.706331 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.706245 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.706331 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.706262 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.706331 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.706295 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.706576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.706335 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.706576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.706404 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.706576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.706449 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fzzp\" (UniqueName: \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-kube-api-access-7fzzp\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.706576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.706509 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.707596 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.707567 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.710095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.708383 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.710095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.708643 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.710095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.708885 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.710095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.709149 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.711907 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.711858 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.712276 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.712250 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.720495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.720457 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.722011 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.721967 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fzzp\" (UniqueName: \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-kube-api-access-7fzzp\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.815928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.815886 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:26.933537 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.933500 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:20:26.933757 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:26.933563 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:20:27.219730 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:20:27.219688 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ab1680_3c8d_48aa_a226_3d420b0d98ad.slice/crio-4d17a8ae0f52941524bb6dcf53fcbab66982af6882f4914fe3867309bfd8c40a WatchSource:0}: Error finding container 4d17a8ae0f52941524bb6dcf53fcbab66982af6882f4914fe3867309bfd8c40a: Status 404 returned error can't find the container with id 4d17a8ae0f52941524bb6dcf53fcbab66982af6882f4914fe3867309bfd8c40a Oct 31 03:20:27.220287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:27.220217 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q"] Oct 31 03:20:27.222417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:27.222381 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:20:27.222521 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:27.222448 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:20:27.222521 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:27.222477 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:20:27.421058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:27.420344 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" event={"ID":"b9ab1680-3c8d-48aa-a226-3d420b0d98ad","Type":"ContainerStarted","Data":"4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8"} Oct 31 03:20:27.421058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:27.420405 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" event={"ID":"b9ab1680-3c8d-48aa-a226-3d420b0d98ad","Type":"ContainerStarted","Data":"4d17a8ae0f52941524bb6dcf53fcbab66982af6882f4914fe3867309bfd8c40a"} Oct 31 03:20:27.497427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:27.496630 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" podStartSLOduration=1.496605572 podStartE2EDuration="1.496605572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:27.445608079 +0000 UTC m=+1509.126134207" watchObservedRunningTime="2026-10-31 03:20:27.496605572 +0000 UTC m=+1509.177131701" Oct 31 03:20:27.821085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:27.816291 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:27.933454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:27.933422 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:20:27.933602 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:27.933478 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:20:28.818586 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:28.818535 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:20:28.819121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:28.818614 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:20:28.933122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:28.933083 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:20:28.933302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:28.933151 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:20:29.825636 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:29.825599 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:20:29.826352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:29.826314 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:20:29.933179 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:29.933142 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:20:29.933494 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:29.933441 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:20:30.817066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:30.816557 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:20:30.817066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:30.816631 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:20:30.940061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:30.935563 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:20:30.940061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:30.935630 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:20:31.816643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:31.816610 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:20:31.816963 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:31.816932 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:20:31.938187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:31.938155 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:31.938476 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:31.938454 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:31.939339 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:31.939317 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:20:32.136893 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:20:32.136800 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:20:32.818453 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:32.818414 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:20:32.818749 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:32.818711 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:20:33.817121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:33.817082 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 03:20:33.817535 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:33.817153 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 03:20:34.391769 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:34.391735 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:20:34.392844 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:34.392819 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:20:34.822539 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:34.822502 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:34.822993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:34.822829 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:34.823649 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:34.823630 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:20:36.095534 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.095427 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt"] Oct 31 03:20:36.104502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.104475 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.112060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.112018 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt"] Oct 31 03:20:36.116078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.116011 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-62bf00\"/\"test-model-62bf00-kserve-self-signed-certs\"" Oct 31 03:20:36.116213 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.116161 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-62bf00\"/\"default-dockercfg-r8bl7\"" Oct 31 03:20:36.127108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.127084 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-62bf00\"/\"kube-root-ca.crt\"" Oct 31 03:20:36.127272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.127114 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-62bf00\"/\"openshift-service-ca.crt\"" Oct 31 03:20:36.247376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.247331 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b473df1-9dae-4fcf-989e-86597c522258-tls-certs\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.247581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.247408 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-dshm\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.247581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.247521 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-home\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.247581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.247548 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-model-cache\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.247581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.247576 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh9bd\" (UniqueName: \"kubernetes.io/projected/5b473df1-9dae-4fcf-989e-86597c522258-kube-api-access-xh9bd\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.247818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.247617 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-tmp-dir\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.349115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.348991 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-dshm\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.349475 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.349448 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-home\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.349640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.349589 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-model-cache\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.349809 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.349790 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh9bd\" (UniqueName: \"kubernetes.io/projected/5b473df1-9dae-4fcf-989e-86597c522258-kube-api-access-xh9bd\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.349959 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.349943 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-tmp-dir\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.350307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.350290 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b473df1-9dae-4fcf-989e-86597c522258-tls-certs\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.352308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.352286 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-model-cache\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.353992 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.353527 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-tmp-dir\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.387090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.371549 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-home\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.387090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.375747 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh9bd\" (UniqueName: \"kubernetes.io/projected/5b473df1-9dae-4fcf-989e-86597c522258-kube-api-access-xh9bd\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.388413 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.388376 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-dshm\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.390158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.390108 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b473df1-9dae-4fcf-989e-86597c522258-tls-certs\") pod \"test-model-62bf00-kserve-c7d74d4cf-tq8nt\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.428908 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.428422 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:36.774827 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:20:36.774797 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b473df1_9dae_4fcf_989e_86597c522258.slice/crio-33c34583f08a5daf9b938a71b4c037440566316b1cc6c74afe61a17caeec1329 WatchSource:0}: Error finding container 33c34583f08a5daf9b938a71b4c037440566316b1cc6c74afe61a17caeec1329: Status 404 returned error can't find the container with id 33c34583f08a5daf9b938a71b4c037440566316b1cc6c74afe61a17caeec1329 Oct 31 03:20:36.776227 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:36.776198 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt"] Oct 31 03:20:37.698277 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:37.698234 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" event={"ID":"5b473df1-9dae-4fcf-989e-86597c522258","Type":"ContainerStarted","Data":"42474901677c078cc046f756302a62f162fc7d504d78f00ff2e6672a0c2faea3"} Oct 31 03:20:37.698727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:37.698287 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" event={"ID":"5b473df1-9dae-4fcf-989e-86597c522258","Type":"ContainerStarted","Data":"33c34583f08a5daf9b938a71b4c037440566316b1cc6c74afe61a17caeec1329"} Oct 31 03:20:37.723486 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:37.723436 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" podStartSLOduration=1.7234220260000002 podStartE2EDuration="1.723422026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:37.719837669 +0000 UTC m=+1519.400363795" watchObservedRunningTime="2026-10-31 03:20:37.723422026 +0000 UTC m=+1519.403948185" Oct 31 03:20:38.439181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:38.439148 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:46.074261 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.074217 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b"] Oct 31 03:20:46.079804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.079785 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.088897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.088711 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b79c13\"/\"kube-root-ca.crt\"" Oct 31 03:20:46.088897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.088805 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b79c13\"/\"openshift-service-ca.crt\"" Oct 31 03:20:46.095163 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.094263 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b79c13\"/\"default-dockercfg-hxsht\"" Oct 31 03:20:46.095163 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.094356 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b79c13\"/\"test-model-b79c13-kserve-self-signed-certs\"" Oct 31 03:20:46.106431 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.106403 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b"] Oct 31 03:20:46.124945 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.124916 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-home\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.125129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.124985 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9th27\" (UniqueName: \"kubernetes.io/projected/814edc80-f564-47c1-a4e5-dc99d746e038-kube-api-access-9th27\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.125238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.125217 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-dshm\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.125445 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.125418 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-tmp-dir\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.125528 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.125459 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814edc80-f564-47c1-a4e5-dc99d746e038-tls-certs\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.125621 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.125606 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-model-cache\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.227472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.227433 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-model-cache\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.227645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.227515 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-home\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.227645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.227573 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9th27\" (UniqueName: \"kubernetes.io/projected/814edc80-f564-47c1-a4e5-dc99d746e038-kube-api-access-9th27\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.227794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.227760 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-dshm\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.227952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.227800 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-tmp-dir\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.228083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.227926 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-model-cache\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.228083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.227983 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814edc80-f564-47c1-a4e5-dc99d746e038-tls-certs\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.228083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.227990 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-home\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.228259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.228170 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-tmp-dir\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.230186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.230159 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-dshm\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.230546 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.230520 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814edc80-f564-47c1-a4e5-dc99d746e038-tls-certs\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.240578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.240559 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9th27\" (UniqueName: \"kubernetes.io/projected/814edc80-f564-47c1-a4e5-dc99d746e038-kube-api-access-9th27\") pod \"test-model-b79c13-kserve-54d8d64bbf-dr42b\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.404371 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.404327 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:46.476059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.464333 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:46.476059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.465773 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:46.758722 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:20:46.758691 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod814edc80_f564_47c1_a4e5_dc99d746e038.slice/crio-818efd1544994fa96e95cc795d7c738c5bdf25e2bd3897c51c850d36b1dd80c8 WatchSource:0}: Error finding container 818efd1544994fa96e95cc795d7c738c5bdf25e2bd3897c51c850d36b1dd80c8: Status 404 returned error can't find the container with id 818efd1544994fa96e95cc795d7c738c5bdf25e2bd3897c51c850d36b1dd80c8 Oct 31 03:20:46.759126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.759102 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b"] Oct 31 03:20:46.809860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.809824 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" event={"ID":"814edc80-f564-47c1-a4e5-dc99d746e038","Type":"ContainerStarted","Data":"818efd1544994fa96e95cc795d7c738c5bdf25e2bd3897c51c850d36b1dd80c8"} Oct 31 03:20:46.822699 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:46.822677 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:20:47.821372 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:47.821331 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" event={"ID":"814edc80-f564-47c1-a4e5-dc99d746e038","Type":"ContainerStarted","Data":"00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7"} Oct 31 03:20:47.850836 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:47.850774 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" podStartSLOduration=1.850756156 podStartE2EDuration="1.850756156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:47.844263331 +0000 UTC m=+1529.524789457" watchObservedRunningTime="2026-10-31 03:20:47.850756156 +0000 UTC m=+1529.531282285" Oct 31 03:20:48.405516 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:48.405472 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:56.406994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.406951 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:56.420061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.420016 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:56.624057 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.621211 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld"] Oct 31 03:20:56.634968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.631693 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld"] Oct 31 03:20:56.634968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.631813 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.732345 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.732250 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.732345 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.732308 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfjk\" (UniqueName: \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-kube-api-access-xgfjk\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.732576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.732410 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.732576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.732495 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.732576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.732546 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.732745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.732594 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.732745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.732692 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.732852 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.732756 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.732852 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.732793 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.834188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834142 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.834381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834195 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfjk\" (UniqueName: \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-kube-api-access-xgfjk\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.834381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834235 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.834381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834296 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.834381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834329 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.834381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834368 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.834671 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834415 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.834671 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834472 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.834671 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834508 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.834916 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834879 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.835084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.834919 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.835168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.835151 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.835238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.835223 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.835696 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.835672 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.837523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.837493 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.837762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.837741 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.848100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.848077 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfjk\" (UniqueName: \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-kube-api-access-xgfjk\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.848226 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.848120 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:56.910214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.910175 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n"] Oct 31 03:20:56.915918 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.915887 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:56.927443 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.927415 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n"] Oct 31 03:20:56.958374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.958343 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:57.000185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:56.999977 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:20:57.005734 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.005607 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:20:57.005851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.005758 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:20:57.042344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.042118 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:57.042344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.042179 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h449f\" (UniqueName: \"kubernetes.io/projected/14e96c53-8917-4efa-b766-c9108d3a0d09-kube-api-access-h449f\") pod \"maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:57.042344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.042312 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:57.153062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.149056 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:57.153062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.149140 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h449f\" (UniqueName: \"kubernetes.io/projected/14e96c53-8917-4efa-b766-c9108d3a0d09-kube-api-access-h449f\") pod \"maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:57.153062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.149311 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:57.153062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.152382 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:57.160515 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.157542 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:57.186388 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.170693 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h449f\" (UniqueName: \"kubernetes.io/projected/14e96c53-8917-4efa-b766-c9108d3a0d09-kube-api-access-h449f\") pod \"maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:57.263664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.263342 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:20:57.308873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.308728 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld"] Oct 31 03:20:57.315406 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:20:57.315375 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc962528e_e5f8_4c4a_a8d1_5220b596fdf4.slice/crio-6041905b1c832ad109f5caf7f8a7e866d23d197c2f291b6add8d192b9300e28d WatchSource:0}: Error finding container 6041905b1c832ad109f5caf7f8a7e866d23d197c2f291b6add8d192b9300e28d: Status 404 returned error can't find the container with id 6041905b1c832ad109f5caf7f8a7e866d23d197c2f291b6add8d192b9300e28d Oct 31 03:20:57.320382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.320346 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:20:57.320502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.320440 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:20:57.320502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.320489 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:20:57.833404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:57.833368 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n"] Oct 31 03:20:58.012792 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:58.012691 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" event={"ID":"14e96c53-8917-4efa-b766-c9108d3a0d09","Type":"ContainerStarted","Data":"473b0e5de1463300c73f5e7d4edb77529b65fd947163c90552ec94b52c44f917"} Oct 31 03:20:58.015676 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:58.015638 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" event={"ID":"c962528e-e5f8-4c4a-a8d1-5220b596fdf4","Type":"ContainerStarted","Data":"57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a"} Oct 31 03:20:58.015833 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:58.015725 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" event={"ID":"c962528e-e5f8-4c4a-a8d1-5220b596fdf4","Type":"ContainerStarted","Data":"6041905b1c832ad109f5caf7f8a7e866d23d197c2f291b6add8d192b9300e28d"} Oct 31 03:20:58.036715 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:58.036681 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj"] Oct 31 03:20:58.041122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:58.041100 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b8712-blwnj"] Oct 31 03:20:58.049342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:58.049001 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" podStartSLOduration=2.048986471 podStartE2EDuration="2.048986471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:58.045485433 +0000 UTC m=+1539.726011560" watchObservedRunningTime="2026-10-31 03:20:58.048986471 +0000 UTC m=+1539.729512599" Oct 31 03:20:58.439643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:58.439610 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fc4a13-d8f5-4488-a51c-8a80b8c8493b" path="/var/lib/kubelet/pods/79fc4a13-d8f5-4488-a51c-8a80b8c8493b/volumes" Oct 31 03:20:58.973061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:58.965793 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:20:58.973061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:58.965966 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 03:20:58.973061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:58.966014 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 03:20:59.030925 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:59.030885 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" event={"ID":"14e96c53-8917-4efa-b766-c9108d3a0d09","Type":"ContainerStarted","Data":"bd585f24706c9b3a5a3963de4fe87475a29959d870111f3765b4409b5cb58710"} Oct 31 03:20:59.104065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:59.101539 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" podStartSLOduration=3.101516342 podStartE2EDuration="3.101516342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:59.054766986 +0000 UTC m=+1540.735293114" watchObservedRunningTime="2026-10-31 03:20:59.101516342 +0000 UTC m=+1540.782042471" Oct 31 03:20:59.967092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:59.965360 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 03:20:59.967092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:20:59.965422 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 03:21:00.041920 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:00.041882 4456 generic.go:356] "Generic (PLEG): container finished" podID="14e96c53-8917-4efa-b766-c9108d3a0d09" containerID="bd585f24706c9b3a5a3963de4fe87475a29959d870111f3765b4409b5cb58710" exitCode=0 Oct 31 03:21:00.042464 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:00.041960 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" event={"ID":"14e96c53-8917-4efa-b766-c9108d3a0d09","Type":"ContainerDied","Data":"bd585f24706c9b3a5a3963de4fe87475a29959d870111f3765b4409b5cb58710"} Oct 31 03:21:00.966182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:00.966141 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 03:21:00.966491 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:00.966461 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 03:21:01.326420 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.326344 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:21:01.421631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.421594 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-openshift-ca\") pod \"14e96c53-8917-4efa-b766-c9108d3a0d09\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " Oct 31 03:21:01.421897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.421876 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e96c53-8917-4efa-b766-c9108d3a0d09-kube-api-access-h449f\" (UniqueName: \"kubernetes.io/projected/14e96c53-8917-4efa-b766-c9108d3a0d09-kube-api-access-h449f\") pod \"14e96c53-8917-4efa-b766-c9108d3a0d09\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " Oct 31 03:21:01.422104 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.422085 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-xks-ca\") pod \"14e96c53-8917-4efa-b766-c9108d3a0d09\" (UID: \"14e96c53-8917-4efa-b766-c9108d3a0d09\") " Oct 31 03:21:01.422289 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.422260 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-openshift-ca" pod "14e96c53-8917-4efa-b766-c9108d3a0d09" (UID: "14e96c53-8917-4efa-b766-c9108d3a0d09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:01.422632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.422613 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:01.424347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.424317 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-xks-ca" pod "14e96c53-8917-4efa-b766-c9108d3a0d09" (UID: "14e96c53-8917-4efa-b766-c9108d3a0d09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:01.424589 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.424560 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e96c53-8917-4efa-b766-c9108d3a0d09-kube-api-access-h449f" pod "14e96c53-8917-4efa-b766-c9108d3a0d09" (UID: "14e96c53-8917-4efa-b766-c9108d3a0d09"). InnerVolumeSpecName "kube-api-access-h449f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:01.525585 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.525548 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e96c53-8917-4efa-b766-c9108d3a0d09-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:01.526022 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.526003 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h449f\" (UniqueName: \"kubernetes.io/projected/14e96c53-8917-4efa-b766-c9108d3a0d09-kube-api-access-h449f\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:01.969066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.965414 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 03:21:01.969066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:01.965476 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 03:21:02.097060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:02.096555 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" event={"ID":"14e96c53-8917-4efa-b766-c9108d3a0d09","Type":"ContainerDied","Data":"473b0e5de1463300c73f5e7d4edb77529b65fd947163c90552ec94b52c44f917"} Oct 31 03:21:02.097060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:02.096656 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="473b0e5de1463300c73f5e7d4edb77529b65fd947163c90552ec94b52c44f917" Oct 31 03:21:02.097060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:02.096725 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n" Oct 31 03:21:02.138058 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:21:02.138012 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:21:02.970060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:02.963572 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 03:21:02.970060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:02.963640 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 03:21:03.889316 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.889276 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9"] Oct 31 03:21:03.890450 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.890423 4456 cpu_manager.go:414] "containerMap: removing container" podUID="14e96c53-8917-4efa-b766-c9108d3a0d09" containerName="revoke-keys" Oct 31 03:21:03.890450 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.890450 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e96c53-8917-4efa-b766-c9108d3a0d09" containerName="revoke-keys" Oct 31 03:21:03.890727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.890711 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e96c53-8917-4efa-b766-c9108d3a0d09" containerName="revoke-keys" Oct 31 03:21:03.895559 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.895535 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:03.902674 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.902200 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:21:03.902674 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.902302 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:21:03.906106 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.906082 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9"] Oct 31 03:21:03.963358 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.963323 4456 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 03:21:03.963558 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.963373 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 03:21:03.990286 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.990245 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245c7ba8-ttxg9\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:03.990764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.990391 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8gd7\" (UniqueName: \"kubernetes.io/projected/4a53a6d6-5983-4739-8bb5-5af445c128cd-kube-api-access-m8gd7\") pod \"maas-api-revoke-sub-245c7ba8-ttxg9\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:03.990764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:03.990533 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245c7ba8-ttxg9\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:04.095068 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.093444 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245c7ba8-ttxg9\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:04.095068 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.093550 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8gd7\" (UniqueName: \"kubernetes.io/projected/4a53a6d6-5983-4739-8bb5-5af445c128cd-kube-api-access-m8gd7\") pod \"maas-api-revoke-sub-245c7ba8-ttxg9\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:04.095068 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.093670 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245c7ba8-ttxg9\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:04.101929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.101894 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245c7ba8-ttxg9\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:04.105688 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.105658 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245c7ba8-ttxg9\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:04.149223 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.149143 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8gd7\" (UniqueName: \"kubernetes.io/projected/4a53a6d6-5983-4739-8bb5-5af445c128cd-kube-api-access-m8gd7\") pod \"maas-api-revoke-sub-245c7ba8-ttxg9\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:04.219327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.219290 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:04.606263 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.606234 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9"] Oct 31 03:21:04.609528 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:21:04.609495 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a53a6d6_5983_4739_8bb5_5af445c128cd.slice/crio-558b760edf14d7f2b82388cc1535903ef46dec774d4244db9283b0e3c8c166a6 WatchSource:0}: Error finding container 558b760edf14d7f2b82388cc1535903ef46dec774d4244db9283b0e3c8c166a6: Status 404 returned error can't find the container with id 558b760edf14d7f2b82388cc1535903ef46dec774d4244db9283b0e3c8c166a6 Oct 31 03:21:04.969713 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.969683 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:21:04.970149 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.970130 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:21:04.971023 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:04.971001 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:21:05.161031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:05.160977 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" event={"ID":"4a53a6d6-5983-4739-8bb5-5af445c128cd","Type":"ContainerStarted","Data":"136b94c3012271e98b5fcf77a7527633a42202c42d4cc049f46cbb10ee4f0b62"} Oct 31 03:21:05.161731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:05.161704 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" event={"ID":"4a53a6d6-5983-4739-8bb5-5af445c128cd","Type":"ContainerStarted","Data":"558b760edf14d7f2b82388cc1535903ef46dec774d4244db9283b0e3c8c166a6"} Oct 31 03:21:05.188597 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:05.188536 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" podStartSLOduration=2.188515889 podStartE2EDuration="2.188515889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:05.184465889 +0000 UTC m=+1546.864992018" watchObservedRunningTime="2026-10-31 03:21:05.188515889 +0000 UTC m=+1546.869042017" Oct 31 03:21:06.173355 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:06.173316 4456 generic.go:356] "Generic (PLEG): container finished" podID="4a53a6d6-5983-4739-8bb5-5af445c128cd" containerID="136b94c3012271e98b5fcf77a7527633a42202c42d4cc049f46cbb10ee4f0b62" exitCode=0 Oct 31 03:21:06.173870 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:06.173388 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" event={"ID":"4a53a6d6-5983-4739-8bb5-5af445c128cd","Type":"ContainerDied","Data":"136b94c3012271e98b5fcf77a7527633a42202c42d4cc049f46cbb10ee4f0b62"} Oct 31 03:21:07.024838 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.024798 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm"] Oct 31 03:21:07.034105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.030772 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc"] Oct 31 03:21:07.541673 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.541645 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:07.707471 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.707436 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-openshift-ca\") pod \"4a53a6d6-5983-4739-8bb5-5af445c128cd\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " Oct 31 03:21:07.707667 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.707496 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-xks-ca\") pod \"4a53a6d6-5983-4739-8bb5-5af445c128cd\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " Oct 31 03:21:07.707667 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.707542 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a53a6d6-5983-4739-8bb5-5af445c128cd-kube-api-access-m8gd7\" (UniqueName: \"kubernetes.io/projected/4a53a6d6-5983-4739-8bb5-5af445c128cd-kube-api-access-m8gd7\") pod \"4a53a6d6-5983-4739-8bb5-5af445c128cd\" (UID: \"4a53a6d6-5983-4739-8bb5-5af445c128cd\") " Oct 31 03:21:07.707832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.707802 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-openshift-ca" pod "4a53a6d6-5983-4739-8bb5-5af445c128cd" (UID: "4a53a6d6-5983-4739-8bb5-5af445c128cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:07.708334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.708291 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:07.711709 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.711660 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a53a6d6-5983-4739-8bb5-5af445c128cd-kube-api-access-m8gd7" pod "4a53a6d6-5983-4739-8bb5-5af445c128cd" (UID: "4a53a6d6-5983-4739-8bb5-5af445c128cd"). InnerVolumeSpecName "kube-api-access-m8gd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:07.711709 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.711697 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-xks-ca" pod "4a53a6d6-5983-4739-8bb5-5af445c128cd" (UID: "4a53a6d6-5983-4739-8bb5-5af445c128cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:07.810008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.809931 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a53a6d6-5983-4739-8bb5-5af445c128cd-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:07.810008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:07.809959 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8gd7\" (UniqueName: \"kubernetes.io/projected/4a53a6d6-5983-4739-8bb5-5af445c128cd-kube-api-access-m8gd7\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:08.196970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.196930 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" Oct 31 03:21:08.197203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.196966 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9" event={"ID":"4a53a6d6-5983-4739-8bb5-5af445c128cd","Type":"ContainerDied","Data":"558b760edf14d7f2b82388cc1535903ef46dec774d4244db9283b0e3c8c166a6"} Oct 31 03:21:08.197203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.197027 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="558b760edf14d7f2b82388cc1535903ef46dec774d4244db9283b0e3c8c166a6" Oct 31 03:21:08.642791 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.642710 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9"] Oct 31 03:21:08.643874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.643846 4456 cpu_manager.go:414] "containerMap: removing container" podUID="4a53a6d6-5983-4739-8bb5-5af445c128cd" containerName="revoke-keys" Oct 31 03:21:08.643874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.643874 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a53a6d6-5983-4739-8bb5-5af445c128cd" containerName="revoke-keys" Oct 31 03:21:08.644129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.644112 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a53a6d6-5983-4739-8bb5-5af445c128cd" containerName="revoke-keys" Oct 31 03:21:08.648446 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.648421 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.666200 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.664975 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9"] Oct 31 03:21:08.730280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.730241 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.730462 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.730294 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.730462 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.730391 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.730462 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.730420 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw4hj\" (UniqueName: \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-kube-api-access-hw4hj\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.730631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.730537 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.730701 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.730665 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.730837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.730814 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.730965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.730947 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d96e62-46ab-4df9-95b8-da022d4fce60-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.731052 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.730992 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.832888 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.832850 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d96e62-46ab-4df9-95b8-da022d4fce60-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.833197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.832913 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.833197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.832997 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.833197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.833025 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.833197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.833089 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.833197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.833117 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4hj\" (UniqueName: \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-kube-api-access-hw4hj\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.833197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.833178 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.833530 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.833262 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.833530 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.833378 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.833917 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.833894 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.834102 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.833942 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d96e62-46ab-4df9-95b8-da022d4fce60-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.834204 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.834186 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.834562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.834536 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.834667 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.834637 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.836870 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.836843 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.841091 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.841069 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.847346 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.847323 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.847470 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.847451 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4hj\" (UniqueName: \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-kube-api-access-hw4hj\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-svpx9\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:08.912508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.912468 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h"] Oct 31 03:21:08.918509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.918484 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:08.926130 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.926102 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h"] Oct 31 03:21:08.935081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.935056 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5t67\" (UniqueName: \"kubernetes.io/projected/0cb55fff-c668-4b41-902b-01c0a86b0e63-kube-api-access-j5t67\") pod \"maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:08.935263 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.935246 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:08.935371 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.935293 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:08.946352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.946307 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:21:08.946352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.946316 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:21:08.969242 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:08.969212 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:09.036580 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.036529 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:09.036771 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.036612 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:09.036858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.036797 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5t67\" (UniqueName: \"kubernetes.io/projected/0cb55fff-c668-4b41-902b-01c0a86b0e63-kube-api-access-j5t67\") pod \"maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:09.038641 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.038521 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:09.040996 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.040970 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:09.054763 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.052835 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5t67\" (UniqueName: \"kubernetes.io/projected/0cb55fff-c668-4b41-902b-01c0a86b0e63-kube-api-access-j5t67\") pod \"maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:09.237427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.237345 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:09.284146 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.284107 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9"] Oct 31 03:21:09.289096 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:21:09.289019 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d96e62_46ab_4df9_95b8_da022d4fce60.slice/crio-4c51bab0e8c4ed27139b609efeddce77a661e3268b4b49d71d61e0c3cf0146e9 WatchSource:0}: Error finding container 4c51bab0e8c4ed27139b609efeddce77a661e3268b4b49d71d61e0c3cf0146e9: Status 404 returned error can't find the container with id 4c51bab0e8c4ed27139b609efeddce77a661e3268b4b49d71d61e0c3cf0146e9 Oct 31 03:21:09.300005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.299952 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:21:09.300148 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.300078 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:21:09.300148 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.300123 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:21:09.542506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:09.542474 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h"] Oct 31 03:21:09.551003 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:21:09.550923 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb55fff_c668_4b41_902b_01c0a86b0e63.slice/crio-9808829a062ef0a1669ba47d2b5a3d5446c6441ef422d80d24fd92bb9c38aacf WatchSource:0}: Error finding container 9808829a062ef0a1669ba47d2b5a3d5446c6441ef422d80d24fd92bb9c38aacf: Status 404 returned error can't find the container with id 9808829a062ef0a1669ba47d2b5a3d5446c6441ef422d80d24fd92bb9c38aacf Oct 31 03:21:10.217114 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:10.217077 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" event={"ID":"81d96e62-46ab-4df9-95b8-da022d4fce60","Type":"ContainerStarted","Data":"98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746"} Oct 31 03:21:10.217582 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:10.217135 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" event={"ID":"81d96e62-46ab-4df9-95b8-da022d4fce60","Type":"ContainerStarted","Data":"4c51bab0e8c4ed27139b609efeddce77a661e3268b4b49d71d61e0c3cf0146e9"} Oct 31 03:21:10.219051 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:10.219017 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" event={"ID":"0cb55fff-c668-4b41-902b-01c0a86b0e63","Type":"ContainerStarted","Data":"516846fd86e688b8c3fe6e0120866d949ab9352edfa75cadaaf460aa31ee98f2"} Oct 31 03:21:10.219177 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:10.219094 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" event={"ID":"0cb55fff-c668-4b41-902b-01c0a86b0e63","Type":"ContainerStarted","Data":"9808829a062ef0a1669ba47d2b5a3d5446c6441ef422d80d24fd92bb9c38aacf"} Oct 31 03:21:10.267068 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:10.264362 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" podStartSLOduration=2.264336238 podStartE2EDuration="2.264336238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:10.24375811 +0000 UTC m=+1551.924284238" watchObservedRunningTime="2026-10-31 03:21:10.264336238 +0000 UTC m=+1551.944862367" Oct 31 03:21:10.287953 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:10.287883 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" podStartSLOduration=2.28786609 podStartE2EDuration="2.28786609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:10.284057039 +0000 UTC m=+1551.964583167" watchObservedRunningTime="2026-10-31 03:21:10.28786609 +0000 UTC m=+1551.968392218" Oct 31 03:21:10.977268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:10.977162 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:10.977457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:10.977309 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:21:10.977457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:10.977359 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" podUID="81d96e62-46ab-4df9-95b8-da022d4fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:21:11.230666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:11.230579 4456 generic.go:356] "Generic (PLEG): container finished" podID="0cb55fff-c668-4b41-902b-01c0a86b0e63" containerID="516846fd86e688b8c3fe6e0120866d949ab9352edfa75cadaaf460aa31ee98f2" exitCode=0 Oct 31 03:21:11.230666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:11.230655 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" event={"ID":"0cb55fff-c668-4b41-902b-01c0a86b0e63","Type":"ContainerDied","Data":"516846fd86e688b8c3fe6e0120866d949ab9352edfa75cadaaf460aa31ee98f2"} Oct 31 03:21:11.972207 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:11.972171 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:21:11.972426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:11.972229 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" podUID="81d96e62-46ab-4df9-95b8-da022d4fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:21:12.026123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.026084 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" podUID="854d04e7-2033-42e6-aeea-1d1fa65ecb7e" containerName="payload-processing" containerID="cri-o://b6b9796d9b3300d33b23e873ff3e943145f18c3a6ff97f6f0f90a87046e2a4b3" gracePeriod=25 Oct 31 03:21:12.031745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.031718 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" podUID="6823bc4b-03bf-4465-a8cb-71e36e6a6b9e" containerName="payload-pre-processing" containerID="cri-o://f39b3d3ed05d4c1387026de771e325edbfe538d93641a5aa711d751b6faf6e16" gracePeriod=25 Oct 31 03:21:12.294084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.290769 4456 generic.go:356] "Generic (PLEG): container finished" podID="854d04e7-2033-42e6-aeea-1d1fa65ecb7e" containerID="b6b9796d9b3300d33b23e873ff3e943145f18c3a6ff97f6f0f90a87046e2a4b3" exitCode=0 Oct 31 03:21:12.294084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.290932 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" event={"ID":"854d04e7-2033-42e6-aeea-1d1fa65ecb7e","Type":"ContainerDied","Data":"b6b9796d9b3300d33b23e873ff3e943145f18c3a6ff97f6f0f90a87046e2a4b3"} Oct 31 03:21:12.294084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.293733 4456 generic.go:356] "Generic (PLEG): container finished" podID="6823bc4b-03bf-4465-a8cb-71e36e6a6b9e" containerID="f39b3d3ed05d4c1387026de771e325edbfe538d93641a5aa711d751b6faf6e16" exitCode=0 Oct 31 03:21:12.294084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.293976 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" event={"ID":"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e","Type":"ContainerDied","Data":"f39b3d3ed05d4c1387026de771e325edbfe538d93641a5aa711d751b6faf6e16"} Oct 31 03:21:12.416411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.415335 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:21:12.523872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.523849 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-kube-api-access-lp6d9\" (UniqueName: \"kubernetes.io/projected/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-kube-api-access-lp6d9\") pod \"854d04e7-2033-42e6-aeea-1d1fa65ecb7e\" (UID: \"854d04e7-2033-42e6-aeea-1d1fa65ecb7e\") " Oct 31 03:21:12.524169 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.524150 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-plugins-config-volume\") pod \"854d04e7-2033-42e6-aeea-1d1fa65ecb7e\" (UID: \"854d04e7-2033-42e6-aeea-1d1fa65ecb7e\") " Oct 31 03:21:12.524790 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.524689 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-plugins-config-volume" pod "854d04e7-2033-42e6-aeea-1d1fa65ecb7e" (UID: "854d04e7-2033-42e6-aeea-1d1fa65ecb7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:12.525145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.525126 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-kube-api-access-lp6d9" pod "854d04e7-2033-42e6-aeea-1d1fa65ecb7e" (UID: "854d04e7-2033-42e6-aeea-1d1fa65ecb7e"). InnerVolumeSpecName "kube-api-access-lp6d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:12.526506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.526482 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp6d9\" (UniqueName: \"kubernetes.io/projected/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-kube-api-access-lp6d9\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:12.526637 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.526624 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854d04e7-2033-42e6-aeea-1d1fa65ecb7e-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:12.543023 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.543004 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:21:12.613428 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.613400 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:12.627541 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.627510 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-plugins-config-volume\") pod \"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e\" (UID: \"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e\") " Oct 31 03:21:12.627706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.627571 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-kube-api-access-b2cq6\" (UniqueName: \"kubernetes.io/projected/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-kube-api-access-b2cq6\") pod \"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e\" (UID: \"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e\") " Oct 31 03:21:12.628488 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.628463 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-plugins-config-volume" pod "6823bc4b-03bf-4465-a8cb-71e36e6a6b9e" (UID: "6823bc4b-03bf-4465-a8cb-71e36e6a6b9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:12.630487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.630457 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-kube-api-access-b2cq6" pod "6823bc4b-03bf-4465-a8cb-71e36e6a6b9e" (UID: "6823bc4b-03bf-4465-a8cb-71e36e6a6b9e"). InnerVolumeSpecName "kube-api-access-b2cq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:12.729615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.729578 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb55fff-c668-4b41-902b-01c0a86b0e63-kube-api-access-j5t67\" (UniqueName: \"kubernetes.io/projected/0cb55fff-c668-4b41-902b-01c0a86b0e63-kube-api-access-j5t67\") pod \"0cb55fff-c668-4b41-902b-01c0a86b0e63\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " Oct 31 03:21:12.729827 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.729811 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-xks-ca\") pod \"0cb55fff-c668-4b41-902b-01c0a86b0e63\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " Oct 31 03:21:12.729893 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.729879 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-openshift-ca\") pod \"0cb55fff-c668-4b41-902b-01c0a86b0e63\" (UID: \"0cb55fff-c668-4b41-902b-01c0a86b0e63\") " Oct 31 03:21:12.730345 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.730315 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-openshift-ca" pod "0cb55fff-c668-4b41-902b-01c0a86b0e63" (UID: "0cb55fff-c668-4b41-902b-01c0a86b0e63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:12.730518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.730500 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:12.730578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.730524 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2cq6\" (UniqueName: \"kubernetes.io/projected/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e-kube-api-access-b2cq6\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:12.730578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.730539 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:12.731833 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.731804 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb55fff-c668-4b41-902b-01c0a86b0e63-kube-api-access-j5t67" pod "0cb55fff-c668-4b41-902b-01c0a86b0e63" (UID: "0cb55fff-c668-4b41-902b-01c0a86b0e63"). InnerVolumeSpecName "kube-api-access-j5t67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:12.731966 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.731945 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-xks-ca" pod "0cb55fff-c668-4b41-902b-01c0a86b0e63" (UID: "0cb55fff-c668-4b41-902b-01c0a86b0e63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:12.863126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.860226 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5t67\" (UniqueName: \"kubernetes.io/projected/0cb55fff-c668-4b41-902b-01c0a86b0e63-kube-api-access-j5t67\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:12.863126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.860268 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb55fff-c668-4b41-902b-01c0a86b0e63-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:12.980061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.972407 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:21:12.980061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:12.972470 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" podUID="81d96e62-46ab-4df9-95b8-da022d4fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:21:13.304720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.304679 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" Oct 31 03:21:13.304720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.304696 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h" event={"ID":"0cb55fff-c668-4b41-902b-01c0a86b0e63","Type":"ContainerDied","Data":"9808829a062ef0a1669ba47d2b5a3d5446c6441ef422d80d24fd92bb9c38aacf"} Oct 31 03:21:13.305199 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.304745 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9808829a062ef0a1669ba47d2b5a3d5446c6441ef422d80d24fd92bb9c38aacf" Oct 31 03:21:13.306502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.306475 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" Oct 31 03:21:13.306640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.306512 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc" event={"ID":"6823bc4b-03bf-4465-a8cb-71e36e6a6b9e","Type":"ContainerDied","Data":"ad864a5ac334aca19aa1b65aaf94e216a14b98f73d89d3651585030296cbe989"} Oct 31 03:21:13.306640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.306572 4456 scope.go:122] "RemoveContainer" containerID="f39b3d3ed05d4c1387026de771e325edbfe538d93641a5aa711d751b6faf6e16" Oct 31 03:21:13.309551 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.309518 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" event={"ID":"854d04e7-2033-42e6-aeea-1d1fa65ecb7e","Type":"ContainerDied","Data":"03086fe0303c2b7e0514a7f63a10537443933c26f94b8e3d18c40ee3bea275ea"} Oct 31 03:21:13.309663 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.309612 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm" Oct 31 03:21:13.322678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.322658 4456 scope.go:122] "RemoveContainer" containerID="b6b9796d9b3300d33b23e873ff3e943145f18c3a6ff97f6f0f90a87046e2a4b3" Oct 31 03:21:13.383519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.383490 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm"] Oct 31 03:21:13.388876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.388845 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fccff5-564d567d8f-xrnzm"] Oct 31 03:21:13.393519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.393369 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc"] Oct 31 03:21:13.398611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.398590 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fccff5-6df5fff59f-n8xjc"] Oct 31 03:21:13.973061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.972357 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:21:13.973061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:13.972415 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" podUID="81d96e62-46ab-4df9-95b8-da022d4fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:21:14.299504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.296278 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6"] Oct 31 03:21:14.299504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.297398 4456 cpu_manager.go:414] "containerMap: removing container" podUID="6823bc4b-03bf-4465-a8cb-71e36e6a6b9e" containerName="payload-pre-processing" Oct 31 03:21:14.299504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.297420 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6823bc4b-03bf-4465-a8cb-71e36e6a6b9e" containerName="payload-pre-processing" Oct 31 03:21:14.299504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.297442 4456 cpu_manager.go:414] "containerMap: removing container" podUID="854d04e7-2033-42e6-aeea-1d1fa65ecb7e" containerName="payload-processing" Oct 31 03:21:14.299504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.297450 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854d04e7-2033-42e6-aeea-1d1fa65ecb7e" containerName="payload-processing" Oct 31 03:21:14.299504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.297506 4456 cpu_manager.go:414] "containerMap: removing container" podUID="0cb55fff-c668-4b41-902b-01c0a86b0e63" containerName="revoke-keys" Oct 31 03:21:14.299504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.297513 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb55fff-c668-4b41-902b-01c0a86b0e63" containerName="revoke-keys" Oct 31 03:21:14.299504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.297693 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb55fff-c668-4b41-902b-01c0a86b0e63" containerName="revoke-keys" Oct 31 03:21:14.299504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.297714 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6823bc4b-03bf-4465-a8cb-71e36e6a6b9e" containerName="payload-pre-processing" Oct 31 03:21:14.299504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.297726 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="854d04e7-2033-42e6-aeea-1d1fa65ecb7e" containerName="payload-processing" Oct 31 03:21:14.314590 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.314564 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6"] Oct 31 03:21:14.315095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.314667 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.323970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.323929 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:21:14.324151 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.323970 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:21:14.395645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.395610 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.395895 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.395877 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfk4\" (UniqueName: \"kubernetes.io/projected/45306927-65b7-4e4f-b86c-b107938ff6f0-kube-api-access-rxfk4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.395976 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.395961 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.455066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.446360 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6823bc4b-03bf-4465-a8cb-71e36e6a6b9e" path="/var/lib/kubelet/pods/6823bc4b-03bf-4465-a8cb-71e36e6a6b9e/volumes" Oct 31 03:21:14.455066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.446965 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854d04e7-2033-42e6-aeea-1d1fa65ecb7e" path="/var/lib/kubelet/pods/854d04e7-2033-42e6-aeea-1d1fa65ecb7e/volumes" Oct 31 03:21:14.497497 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.497456 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.497717 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.497520 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.497986 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.497958 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfk4\" (UniqueName: \"kubernetes.io/projected/45306927-65b7-4e4f-b86c-b107938ff6f0-kube-api-access-rxfk4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.498317 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.498291 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.507943 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.507919 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.512998 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.512970 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfk4\" (UniqueName: \"kubernetes.io/projected/45306927-65b7-4e4f-b86c-b107938ff6f0-kube-api-access-rxfk4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.632690 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.632608 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:14.974727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.974684 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:21:14.974937 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.974751 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" podUID="81d96e62-46ab-4df9-95b8-da022d4fce60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:21:14.978214 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:14.978186 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6"] Oct 31 03:21:14.979827 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:21:14.979798 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45306927_65b7_4e4f_b86c_b107938ff6f0.slice/crio-3f8abf1f36872974f8e9d571ae5dd3c1010329004a2dfd218b9dcff1d51f9261 WatchSource:0}: Error finding container 3f8abf1f36872974f8e9d571ae5dd3c1010329004a2dfd218b9dcff1d51f9261: Status 404 returned error can't find the container with id 3f8abf1f36872974f8e9d571ae5dd3c1010329004a2dfd218b9dcff1d51f9261 Oct 31 03:21:15.036518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:15.036479 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5"] Oct 31 03:21:15.041742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:15.041715 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83a30d1-dtfb5"] Oct 31 03:21:15.339659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:15.339570 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" event={"ID":"45306927-65b7-4e4f-b86c-b107938ff6f0","Type":"ContainerStarted","Data":"3f8abf1f36872974f8e9d571ae5dd3c1010329004a2dfd218b9dcff1d51f9261"} Oct 31 03:21:15.975503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:15.975470 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:15.975964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:15.975935 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:15.976744 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:15.976693 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:21:16.351894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:16.351804 4456 generic.go:356] "Generic (PLEG): container finished" podID="45306927-65b7-4e4f-b86c-b107938ff6f0" containerID="66edb7929cc5e7894d330735c915030e5e7052947c24f197314553bfd7bc3780" exitCode=0 Oct 31 03:21:16.352539 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:16.351880 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" event={"ID":"45306927-65b7-4e4f-b86c-b107938ff6f0","Type":"ContainerDied","Data":"66edb7929cc5e7894d330735c915030e5e7052947c24f197314553bfd7bc3780"} Oct 31 03:21:16.437809 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:16.437779 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175a71f2-f2bd-4139-a641-1e7d357b04e0" path="/var/lib/kubelet/pods/175a71f2-f2bd-4139-a641-1e7d357b04e0/volumes" Oct 31 03:21:17.554597 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:17.554570 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:17.663239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:17.653408 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45306927-65b7-4e4f-b86c-b107938ff6f0-kube-api-access-rxfk4\" (UniqueName: \"kubernetes.io/projected/45306927-65b7-4e4f-b86c-b107938ff6f0-kube-api-access-rxfk4\") pod \"45306927-65b7-4e4f-b86c-b107938ff6f0\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " Oct 31 03:21:17.663239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:17.659838 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-openshift-ca\") pod \"45306927-65b7-4e4f-b86c-b107938ff6f0\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " Oct 31 03:21:17.663239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:17.660373 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-openshift-ca" pod "45306927-65b7-4e4f-b86c-b107938ff6f0" (UID: "45306927-65b7-4e4f-b86c-b107938ff6f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:17.663239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:17.660785 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-xks-ca\") pod \"45306927-65b7-4e4f-b86c-b107938ff6f0\" (UID: \"45306927-65b7-4e4f-b86c-b107938ff6f0\") " Oct 31 03:21:17.663239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:17.661942 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:17.682326 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:17.681838 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-xks-ca" pod "45306927-65b7-4e4f-b86c-b107938ff6f0" (UID: "45306927-65b7-4e4f-b86c-b107938ff6f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:17.683321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:17.683287 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45306927-65b7-4e4f-b86c-b107938ff6f0-kube-api-access-rxfk4" pod "45306927-65b7-4e4f-b86c-b107938ff6f0" (UID: "45306927-65b7-4e4f-b86c-b107938ff6f0"). InnerVolumeSpecName "kube-api-access-rxfk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:17.763296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:17.763257 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxfk4\" (UniqueName: \"kubernetes.io/projected/45306927-65b7-4e4f-b86c-b107938ff6f0-kube-api-access-rxfk4\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:17.763476 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:17.763309 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45306927-65b7-4e4f-b86c-b107938ff6f0-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:18.380204 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:18.380166 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" Oct 31 03:21:18.380378 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:18.380175 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6" event={"ID":"45306927-65b7-4e4f-b86c-b107938ff6f0","Type":"ContainerDied","Data":"3f8abf1f36872974f8e9d571ae5dd3c1010329004a2dfd218b9dcff1d51f9261"} Oct 31 03:21:18.380378 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:18.380329 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f8abf1f36872974f8e9d571ae5dd3c1010329004a2dfd218b9dcff1d51f9261" Oct 31 03:21:19.220882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:19.220842 4456 scope.go:122] "RemoveContainer" containerID="0d25bc2e76cc6e455f8d0e273d54de693fba7185155052aad19905397480f38c" Oct 31 03:21:21.227348 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.227305 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng"] Oct 31 03:21:21.229143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.229111 4456 cpu_manager.go:414] "containerMap: removing container" podUID="45306927-65b7-4e4f-b86c-b107938ff6f0" containerName="revoke-keys" Oct 31 03:21:21.229143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.229141 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45306927-65b7-4e4f-b86c-b107938ff6f0" containerName="revoke-keys" Oct 31 03:21:21.229602 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.229579 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="45306927-65b7-4e4f-b86c-b107938ff6f0" containerName="revoke-keys" Oct 31 03:21:21.237702 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.237675 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.239014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.238986 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng"] Oct 31 03:21:21.242470 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.242441 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:21:21.244546 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.244524 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:21:21.245525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.245502 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f"] Oct 31 03:21:21.265523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.265497 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.278698 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.278108 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f"] Oct 31 03:21:21.324892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.324859 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887303fd-s6k6f\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.325080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.324918 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-971a0b35-xz7ng\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.325080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.325029 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgtk8\" (UniqueName: \"kubernetes.io/projected/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-kube-api-access-hgtk8\") pod \"maas-api-revoke-sub-971a0b35-xz7ng\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.325183 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.325164 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-769m4\" (UniqueName: \"kubernetes.io/projected/b4cffa9e-6a33-458c-b16c-203213456d12-kube-api-access-769m4\") pod \"maas-api-revoke-sub-887303fd-s6k6f\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.325301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.325286 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-971a0b35-xz7ng\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.325339 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.325309 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887303fd-s6k6f\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.427222 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.427176 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887303fd-s6k6f\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.427417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.427262 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-971a0b35-xz7ng\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.427417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.427331 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtk8\" (UniqueName: \"kubernetes.io/projected/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-kube-api-access-hgtk8\") pod \"maas-api-revoke-sub-971a0b35-xz7ng\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.427417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.427390 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-769m4\" (UniqueName: \"kubernetes.io/projected/b4cffa9e-6a33-458c-b16c-203213456d12-kube-api-access-769m4\") pod \"maas-api-revoke-sub-887303fd-s6k6f\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.427650 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.427535 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-971a0b35-xz7ng\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.427650 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.427560 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887303fd-s6k6f\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.428007 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.427979 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887303fd-s6k6f\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.428295 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.428273 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-971a0b35-xz7ng\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.429797 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.429771 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-971a0b35-xz7ng\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.429879 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.429804 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887303fd-s6k6f\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.440421 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.440395 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-769m4\" (UniqueName: \"kubernetes.io/projected/b4cffa9e-6a33-458c-b16c-203213456d12-kube-api-access-769m4\") pod \"maas-api-revoke-sub-887303fd-s6k6f\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.440522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.440439 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtk8\" (UniqueName: \"kubernetes.io/projected/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-kube-api-access-hgtk8\") pod \"maas-api-revoke-sub-971a0b35-xz7ng\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.570056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.569958 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:21.590235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.590205 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:21.933652 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:21:21.933624 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bc60b4_98d7_4005_97ea_1cadc44f7aab.slice/crio-803ac516e02b1c4b5cb72232bad307a98178acd7d807a0c1dad937169ddea96e WatchSource:0}: Error finding container 803ac516e02b1c4b5cb72232bad307a98178acd7d807a0c1dad937169ddea96e: Status 404 returned error can't find the container with id 803ac516e02b1c4b5cb72232bad307a98178acd7d807a0c1dad937169ddea96e Oct 31 03:21:21.935165 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.935139 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng"] Oct 31 03:21:21.971519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:21.971488 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f"] Oct 31 03:21:21.977593 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:21:21.977568 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cffa9e_6a33_458c_b16c_203213456d12.slice/crio-09615efe2eaa129aed78ca36dc7b5bb592ae7add8f0ded0121222700fbdf018c WatchSource:0}: Error finding container 09615efe2eaa129aed78ca36dc7b5bb592ae7add8f0ded0121222700fbdf018c: Status 404 returned error can't find the container with id 09615efe2eaa129aed78ca36dc7b5bb592ae7add8f0ded0121222700fbdf018c Oct 31 03:21:22.499304 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:22.499265 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" event={"ID":"b4cffa9e-6a33-458c-b16c-203213456d12","Type":"ContainerStarted","Data":"7239aaf2e9229116dc4f9f227522bfcf96e7c0cea7c7c0b0d3da2cfc97d3b1af"} Oct 31 03:21:22.499925 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:22.499333 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" event={"ID":"b4cffa9e-6a33-458c-b16c-203213456d12","Type":"ContainerStarted","Data":"09615efe2eaa129aed78ca36dc7b5bb592ae7add8f0ded0121222700fbdf018c"} Oct 31 03:21:22.502674 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:22.502645 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" event={"ID":"e4bc60b4-98d7-4005-97ea-1cadc44f7aab","Type":"ContainerStarted","Data":"1daed2f99f225c05fe0d4f9541d5a2d848a169bf3a7829d16e08ff52187cae10"} Oct 31 03:21:22.502805 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:22.502706 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" event={"ID":"e4bc60b4-98d7-4005-97ea-1cadc44f7aab","Type":"ContainerStarted","Data":"803ac516e02b1c4b5cb72232bad307a98178acd7d807a0c1dad937169ddea96e"} Oct 31 03:21:22.549181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:22.549120 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" podStartSLOduration=1.549104402 podStartE2EDuration="1.549104402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:22.54891066 +0000 UTC m=+1564.229436788" watchObservedRunningTime="2026-10-31 03:21:22.549104402 +0000 UTC m=+1564.229630530" Oct 31 03:21:22.576022 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:22.575966 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" podStartSLOduration=1.5759514289999998 podStartE2EDuration="1.575951429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:22.57338713 +0000 UTC m=+1564.253913258" watchObservedRunningTime="2026-10-31 03:21:22.575951429 +0000 UTC m=+1564.256477553" Oct 31 03:21:23.514860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.514821 4456 generic.go:356] "Generic (PLEG): container finished" podID="b4cffa9e-6a33-458c-b16c-203213456d12" containerID="7239aaf2e9229116dc4f9f227522bfcf96e7c0cea7c7c0b0d3da2cfc97d3b1af" exitCode=0 Oct 31 03:21:23.515368 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.514870 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" event={"ID":"b4cffa9e-6a33-458c-b16c-203213456d12","Type":"ContainerDied","Data":"7239aaf2e9229116dc4f9f227522bfcf96e7c0cea7c7c0b0d3da2cfc97d3b1af"} Oct 31 03:21:23.517024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.516997 4456 generic.go:356] "Generic (PLEG): container finished" podID="e4bc60b4-98d7-4005-97ea-1cadc44f7aab" containerID="1daed2f99f225c05fe0d4f9541d5a2d848a169bf3a7829d16e08ff52187cae10" exitCode=0 Oct 31 03:21:23.517181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.517157 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" event={"ID":"e4bc60b4-98d7-4005-97ea-1cadc44f7aab","Type":"ContainerDied","Data":"1daed2f99f225c05fe0d4f9541d5a2d848a169bf3a7829d16e08ff52187cae10"} Oct 31 03:21:23.649910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.649868 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp"] Oct 31 03:21:23.658087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.658018 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.679973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.679940 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp"] Oct 31 03:21:23.765874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.765785 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmlgx\" (UniqueName: \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-kube-api-access-mmlgx\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.766082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.765984 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.766082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.766054 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.766182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.766150 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.766226 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.766179 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.766317 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.766301 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.766418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.766400 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.766466 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.766449 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.766510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.766496 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.868470 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.868425 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.868677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.868554 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.868677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.868595 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.868677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.868619 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.868677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.868669 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmlgx\" (UniqueName: \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-kube-api-access-mmlgx\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.868940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.868810 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.868940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.868859 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.868940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.868901 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.868940 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.868927 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.869181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.868952 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.869241 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.869229 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.869470 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.869442 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.869563 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.869497 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.869563 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.869520 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.871364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.871340 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.871594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.871575 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.881934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.881912 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.882144 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.882121 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmlgx\" (UniqueName: \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-kube-api-access-mmlgx\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:23.985803 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:23.985769 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:24.284458 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:24.284425 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp"] Oct 31 03:21:24.285361 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:24.285321 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:21:24.285472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:24.285408 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:21:24.285472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:24.285451 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:21:24.424909 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:24.423099 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw"] Oct 31 03:21:24.471108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:24.465556 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v"] Oct 31 03:21:24.543956 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:24.543915 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" event={"ID":"927cee97-88f9-4e3f-bc1f-10ffc85658ac","Type":"ContainerStarted","Data":"2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda"} Oct 31 03:21:24.544568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:24.543982 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" event={"ID":"927cee97-88f9-4e3f-bc1f-10ffc85658ac","Type":"ContainerStarted","Data":"330be89ada2b13dec3a4a610e9e747ea1b31394f56dd54b97162e4d4f473c8a5"} Oct 31 03:21:24.588814 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:24.588679 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" podStartSLOduration=1.5886610270000001 podStartE2EDuration="1.588661027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:24.587237999 +0000 UTC m=+1566.267764176" watchObservedRunningTime="2026-10-31 03:21:24.588661027 +0000 UTC m=+1566.269187163" Oct 31 03:21:24.988416 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:24.988379 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:25.021126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.021103 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:25.063266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.063234 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:25.177468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.177312 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cffa9e-6a33-458c-b16c-203213456d12-kube-api-access-769m4\" (UniqueName: \"kubernetes.io/projected/b4cffa9e-6a33-458c-b16c-203213456d12-kube-api-access-769m4\") pod \"b4cffa9e-6a33-458c-b16c-203213456d12\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " Oct 31 03:21:25.177655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.177565 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-xks-ca\") pod \"b4cffa9e-6a33-458c-b16c-203213456d12\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " Oct 31 03:21:25.177789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.177770 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-openshift-ca\") pod \"b4cffa9e-6a33-458c-b16c-203213456d12\" (UID: \"b4cffa9e-6a33-458c-b16c-203213456d12\") " Oct 31 03:21:25.177875 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.177860 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-xks-ca\") pod \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " Oct 31 03:21:25.177935 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.177905 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-openshift-ca\") pod \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " Oct 31 03:21:25.177995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.177969 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-kube-api-access-hgtk8\" (UniqueName: \"kubernetes.io/projected/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-kube-api-access-hgtk8\") pod \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\" (UID: \"e4bc60b4-98d7-4005-97ea-1cadc44f7aab\") " Oct 31 03:21:25.180406 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.180364 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-openshift-ca" pod "e4bc60b4-98d7-4005-97ea-1cadc44f7aab" (UID: "e4bc60b4-98d7-4005-97ea-1cadc44f7aab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:25.181468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.181425 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-openshift-ca" pod "b4cffa9e-6a33-458c-b16c-203213456d12" (UID: "b4cffa9e-6a33-458c-b16c-203213456d12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:25.182403 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.182369 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cffa9e-6a33-458c-b16c-203213456d12-kube-api-access-769m4" pod "b4cffa9e-6a33-458c-b16c-203213456d12" (UID: "b4cffa9e-6a33-458c-b16c-203213456d12"). InnerVolumeSpecName "kube-api-access-769m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:25.184194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.184160 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-xks-ca" pod "e4bc60b4-98d7-4005-97ea-1cadc44f7aab" (UID: "e4bc60b4-98d7-4005-97ea-1cadc44f7aab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:25.185262 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.185220 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-xks-ca" pod "b4cffa9e-6a33-458c-b16c-203213456d12" (UID: "b4cffa9e-6a33-458c-b16c-203213456d12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:25.191838 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.191811 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-kube-api-access-hgtk8" pod "e4bc60b4-98d7-4005-97ea-1cadc44f7aab" (UID: "e4bc60b4-98d7-4005-97ea-1cadc44f7aab"). InnerVolumeSpecName "kube-api-access-hgtk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:25.280274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.280238 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:25.280274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.280273 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cffa9e-6a33-458c-b16c-203213456d12-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:25.280274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.280283 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:25.280507 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.280292 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:25.280507 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.280301 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgtk8\" (UniqueName: \"kubernetes.io/projected/e4bc60b4-98d7-4005-97ea-1cadc44f7aab-kube-api-access-hgtk8\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:25.280507 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.280310 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-769m4\" (UniqueName: \"kubernetes.io/projected/b4cffa9e-6a33-458c-b16c-203213456d12-kube-api-access-769m4\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:25.558119 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.557240 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" event={"ID":"b4cffa9e-6a33-458c-b16c-203213456d12","Type":"ContainerDied","Data":"09615efe2eaa129aed78ca36dc7b5bb592ae7add8f0ded0121222700fbdf018c"} Oct 31 03:21:25.558119 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.557309 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09615efe2eaa129aed78ca36dc7b5bb592ae7add8f0ded0121222700fbdf018c" Oct 31 03:21:25.558119 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.557390 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f" Oct 31 03:21:25.575135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.574910 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" Oct 31 03:21:25.586132 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.586087 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng" event={"ID":"e4bc60b4-98d7-4005-97ea-1cadc44f7aab","Type":"ContainerDied","Data":"803ac516e02b1c4b5cb72232bad307a98178acd7d807a0c1dad937169ddea96e"} Oct 31 03:21:25.586362 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.586161 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="803ac516e02b1c4b5cb72232bad307a98178acd7d807a0c1dad937169ddea96e" Oct 31 03:21:25.998301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.998208 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:21:25.998301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:25.998288 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:21:26.208631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:26.208595 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g"] Oct 31 03:21:26.208816 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:26.208794 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" containerID="cri-o://67b00222b91788090798f50a5561e588df911e1bb25e47bdd45d8dcec276d5c5" gracePeriod=30 Oct 31 03:21:26.214808 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:26.214780 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6"] Oct 31 03:21:26.214970 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:26.214945 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" containerID="cri-o://610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4" gracePeriod=30 Oct 31 03:21:26.225441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:26.225251 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b"] Oct 31 03:21:26.225584 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:26.225555 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" containerID="cri-o://8fc96c5e205efbe0f587544fb0508534febab2d7a456edbcbe539b5e1debcdbd" gracePeriod=30 Oct 31 03:21:27.000464 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:27.000424 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:21:27.001114 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:27.001078 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:21:27.986162 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:27.986129 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:21:27.986354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:27.986183 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:21:28.986278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:28.986239 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:21:28.986773 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:28.986305 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:21:29.424281 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:29.424236 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" podUID="e0d91df2-e63c-4fd5-ad16-f45a30251fbb" containerName="payload-processing" containerID="cri-o://75411ede2639c94b6b132b65904fe4513c8f0f82ee2b59c1fc9e37ad98c24fa6" gracePeriod=25 Oct 31 03:21:29.476060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:29.471098 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" podUID="c02eb473-9cd5-4595-926d-e17429e58a8c" containerName="payload-pre-processing" containerID="cri-o://7d99331efa3798c3f3dca169e1052a59eafef432e0967137c7af692ad0eeb29a" gracePeriod=25 Oct 31 03:21:29.621126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:29.621090 4456 generic.go:356] "Generic (PLEG): container finished" podID="e0d91df2-e63c-4fd5-ad16-f45a30251fbb" containerID="75411ede2639c94b6b132b65904fe4513c8f0f82ee2b59c1fc9e37ad98c24fa6" exitCode=0 Oct 31 03:21:29.621327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:29.621159 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" event={"ID":"e0d91df2-e63c-4fd5-ad16-f45a30251fbb","Type":"ContainerDied","Data":"75411ede2639c94b6b132b65904fe4513c8f0f82ee2b59c1fc9e37ad98c24fa6"} Oct 31 03:21:29.623749 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:29.623639 4456 generic.go:356] "Generic (PLEG): container finished" podID="c02eb473-9cd5-4595-926d-e17429e58a8c" containerID="7d99331efa3798c3f3dca169e1052a59eafef432e0967137c7af692ad0eeb29a" exitCode=0 Oct 31 03:21:29.623886 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:29.623789 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" event={"ID":"c02eb473-9cd5-4595-926d-e17429e58a8c","Type":"ContainerDied","Data":"7d99331efa3798c3f3dca169e1052a59eafef432e0967137c7af692ad0eeb29a"} Oct 31 03:21:29.893681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:29.893654 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:21:29.918109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:29.918077 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:21:29.992064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:29.990510 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:21:29.992064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:29.990572 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:21:30.008216 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.008141 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-plugins-config-volume\") pod \"e0d91df2-e63c-4fd5-ad16-f45a30251fbb\" (UID: \"e0d91df2-e63c-4fd5-ad16-f45a30251fbb\") " Oct 31 03:21:30.008368 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.008243 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02eb473-9cd5-4595-926d-e17429e58a8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02eb473-9cd5-4595-926d-e17429e58a8c-plugins-config-volume\") pod \"c02eb473-9cd5-4595-926d-e17429e58a8c\" (UID: \"c02eb473-9cd5-4595-926d-e17429e58a8c\") " Oct 31 03:21:30.008407 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.008393 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02eb473-9cd5-4595-926d-e17429e58a8c-kube-api-access-pc484\" (UniqueName: \"kubernetes.io/projected/c02eb473-9cd5-4595-926d-e17429e58a8c-kube-api-access-pc484\") pod \"c02eb473-9cd5-4595-926d-e17429e58a8c\" (UID: \"c02eb473-9cd5-4595-926d-e17429e58a8c\") " Oct 31 03:21:30.008461 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.008451 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-kube-api-access-b6h9v\" (UniqueName: \"kubernetes.io/projected/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-kube-api-access-b6h9v\") pod \"e0d91df2-e63c-4fd5-ad16-f45a30251fbb\" (UID: \"e0d91df2-e63c-4fd5-ad16-f45a30251fbb\") " Oct 31 03:21:30.008588 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.008566 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-plugins-config-volume" pod "e0d91df2-e63c-4fd5-ad16-f45a30251fbb" (UID: "e0d91df2-e63c-4fd5-ad16-f45a30251fbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:30.008969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.008935 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02eb473-9cd5-4595-926d-e17429e58a8c-plugins-config-volume" pod "c02eb473-9cd5-4595-926d-e17429e58a8c" (UID: "c02eb473-9cd5-4595-926d-e17429e58a8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:30.009758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.009728 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:30.009864 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.009761 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02eb473-9cd5-4595-926d-e17429e58a8c-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:30.010898 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.010872 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02eb473-9cd5-4595-926d-e17429e58a8c-kube-api-access-pc484" pod "c02eb473-9cd5-4595-926d-e17429e58a8c" (UID: "c02eb473-9cd5-4595-926d-e17429e58a8c"). InnerVolumeSpecName "kube-api-access-pc484". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:30.011081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.011055 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-kube-api-access-b6h9v" pod "e0d91df2-e63c-4fd5-ad16-f45a30251fbb" (UID: "e0d91df2-e63c-4fd5-ad16-f45a30251fbb"). InnerVolumeSpecName "kube-api-access-b6h9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:30.120063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.113499 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc484\" (UniqueName: \"kubernetes.io/projected/c02eb473-9cd5-4595-926d-e17429e58a8c-kube-api-access-pc484\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:30.120063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.113546 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6h9v\" (UniqueName: \"kubernetes.io/projected/e0d91df2-e63c-4fd5-ad16-f45a30251fbb-kube-api-access-b6h9v\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:30.634934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.634889 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" event={"ID":"e0d91df2-e63c-4fd5-ad16-f45a30251fbb","Type":"ContainerDied","Data":"fce8f177f8e801f5e499808a5eed3648c1b7ecf1003018385c94d6c7622d28e3"} Oct 31 03:21:30.635145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.634966 4456 scope.go:122] "RemoveContainer" containerID="75411ede2639c94b6b132b65904fe4513c8f0f82ee2b59c1fc9e37ad98c24fa6" Oct 31 03:21:30.635145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.634906 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw" Oct 31 03:21:30.636812 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.636788 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" event={"ID":"c02eb473-9cd5-4595-926d-e17429e58a8c","Type":"ContainerDied","Data":"ad3f26d04688d3e36f1e2608c591a32f31b6900404f3614ed245ad51705e6f68"} Oct 31 03:21:30.636923 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.636908 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v" Oct 31 03:21:30.675306 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.674116 4456 scope.go:122] "RemoveContainer" containerID="7d99331efa3798c3f3dca169e1052a59eafef432e0967137c7af692ad0eeb29a" Oct 31 03:21:30.695023 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.694950 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw"] Oct 31 03:21:30.702101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.702022 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-fd2e0b-6684c46c5f-jp2lw"] Oct 31 03:21:30.710655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.710622 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v"] Oct 31 03:21:30.715349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.715326 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-fd2e0b-674d8c5c59-mgj9v"] Oct 31 03:21:30.990210 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.990110 4456 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:21:30.993949 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:30.990444 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 03:21:31.604624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.592639 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:21:31.661095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.659678 4456 generic.go:356] "Generic (PLEG): container finished" podID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerID="610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4" exitCode=0 Oct 31 03:21:31.661095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.659849 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" event={"ID":"7526b30d-815e-4669-808d-2bf77dbfbc8a","Type":"ContainerDied","Data":"610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4"} Oct 31 03:21:31.661095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.659927 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" event={"ID":"7526b30d-815e-4669-808d-2bf77dbfbc8a","Type":"ContainerDied","Data":"af663ed61b52af130b9713ab11f1fb7b6d818ae4767d150bc902f41680e392b1"} Oct 31 03:21:31.661095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.659977 4456 scope.go:122] "RemoveContainer" containerID="610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4" Oct 31 03:21:31.661095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.660144 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6" Oct 31 03:21:31.665608 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.665580 4456 generic.go:356] "Generic (PLEG): container finished" podID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerID="8fc96c5e205efbe0f587544fb0508534febab2d7a456edbcbe539b5e1debcdbd" exitCode=0 Oct 31 03:21:31.665919 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.665651 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" event={"ID":"f25526fb-70e1-44f2-8157-c4648fc91e9a","Type":"ContainerDied","Data":"8fc96c5e205efbe0f587544fb0508534febab2d7a456edbcbe539b5e1debcdbd"} Oct 31 03:21:31.672758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.668999 4456 generic.go:356] "Generic (PLEG): container finished" podID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerID="67b00222b91788090798f50a5561e588df911e1bb25e47bdd45d8dcec276d5c5" exitCode=0 Oct 31 03:21:31.672758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.669247 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" event={"ID":"0478a844-cbe2-41b0-a4fe-78fcef28831d","Type":"ContainerDied","Data":"67b00222b91788090798f50a5561e588df911e1bb25e47bdd45d8dcec276d5c5"} Oct 31 03:21:31.672758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.669307 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" event={"ID":"0478a844-cbe2-41b0-a4fe-78fcef28831d","Type":"ContainerDied","Data":"97af4a15c6833b893fbadb64c3995d64e0b77eb7033479ca80ce0500b6f737bf"} Oct 31 03:21:31.672758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.669353 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97af4a15c6833b893fbadb64c3995d64e0b77eb7033479ca80ce0500b6f737bf" Oct 31 03:21:31.686226 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.686203 4456 scope.go:122] "RemoveContainer" containerID="610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4" Oct 31 03:21:31.686968 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:21:31.686936 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4\": container with ID starting with 610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4 not found: ID does not exist" containerID="610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4" Oct 31 03:21:31.687075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.686978 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4"} err="failed to get container status \"610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4\": rpc error: code = NotFound desc = could not find container \"610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4\": container with ID starting with 610272b93ba02670e7d7fe21b5e3fa5340efb51f13bc7f41734c58a4215ebcf4 not found: ID does not exist" Oct 31 03:21:31.696616 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.696602 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:21:31.700587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.700572 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:21:31.705789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.705629 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7526b30d-815e-4669-808d-2bf77dbfbc8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7526b30d-815e-4669-808d-2bf77dbfbc8a-istiod-ca-cert\") pod \"7526b30d-815e-4669-808d-2bf77dbfbc8a\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " Oct 31 03:21:31.705789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.705743 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-envoy\") pod \"7526b30d-815e-4669-808d-2bf77dbfbc8a\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " Oct 31 03:21:31.705789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.705782 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-certs\") pod \"7526b30d-815e-4669-808d-2bf77dbfbc8a\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " Oct 31 03:21:31.706123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.705880 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-kube-api-access-5ftvf\" (UniqueName: \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-kube-api-access-5ftvf\") pod \"7526b30d-815e-4669-808d-2bf77dbfbc8a\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " Oct 31 03:21:31.706123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.705950 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-credential-socket\") pod \"7526b30d-815e-4669-808d-2bf77dbfbc8a\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " Oct 31 03:21:31.706123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.705971 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-token\" (UniqueName: \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-token\") pod \"7526b30d-815e-4669-808d-2bf77dbfbc8a\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " Oct 31 03:21:31.706123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.705993 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-podinfo\") pod \"7526b30d-815e-4669-808d-2bf77dbfbc8a\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " Oct 31 03:21:31.706123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.706058 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-data\") pod \"7526b30d-815e-4669-808d-2bf77dbfbc8a\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " Oct 31 03:21:31.706123 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.706081 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-socket\") pod \"7526b30d-815e-4669-808d-2bf77dbfbc8a\" (UID: \"7526b30d-815e-4669-808d-2bf77dbfbc8a\") " Oct 31 03:21:31.706809 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.706784 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-socket" pod "7526b30d-815e-4669-808d-2bf77dbfbc8a" (UID: "7526b30d-815e-4669-808d-2bf77dbfbc8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.707576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.707546 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-credential-socket" pod "7526b30d-815e-4669-808d-2bf77dbfbc8a" (UID: "7526b30d-815e-4669-808d-2bf77dbfbc8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.707664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.707638 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-data" pod "7526b30d-815e-4669-808d-2bf77dbfbc8a" (UID: "7526b30d-815e-4669-808d-2bf77dbfbc8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.707739 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.707691 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-certs" pod "7526b30d-815e-4669-808d-2bf77dbfbc8a" (UID: "7526b30d-815e-4669-808d-2bf77dbfbc8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.708148 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.708122 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7526b30d-815e-4669-808d-2bf77dbfbc8a-istiod-ca-cert" pod "7526b30d-815e-4669-808d-2bf77dbfbc8a" (UID: "7526b30d-815e-4669-808d-2bf77dbfbc8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:31.709273 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.709250 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-podinfo" pod "7526b30d-815e-4669-808d-2bf77dbfbc8a" (UID: "7526b30d-815e-4669-808d-2bf77dbfbc8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:31.709596 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.709546 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-envoy" pod "7526b30d-815e-4669-808d-2bf77dbfbc8a" (UID: "7526b30d-815e-4669-808d-2bf77dbfbc8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.710294 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.710252 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-kube-api-access-5ftvf" pod "7526b30d-815e-4669-808d-2bf77dbfbc8a" (UID: "7526b30d-815e-4669-808d-2bf77dbfbc8a"). InnerVolumeSpecName "kube-api-access-5ftvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:31.710404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.710376 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-token" pod "7526b30d-815e-4669-808d-2bf77dbfbc8a" (UID: "7526b30d-815e-4669-808d-2bf77dbfbc8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:31.807789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.807703 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-podinfo\") pod \"0478a844-cbe2-41b0-a4fe-78fcef28831d\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " Oct 31 03:21:31.807789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.807762 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-token\" (UniqueName: \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-token\") pod \"f25526fb-70e1-44f2-8157-c4648fc91e9a\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " Oct 31 03:21:31.807977 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.807813 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-credential-socket\") pod \"0478a844-cbe2-41b0-a4fe-78fcef28831d\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " Oct 31 03:21:31.807977 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.807946 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-certs\") pod \"f25526fb-70e1-44f2-8157-c4648fc91e9a\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " Oct 31 03:21:31.808092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808024 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0478a844-cbe2-41b0-a4fe-78fcef28831d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0478a844-cbe2-41b0-a4fe-78fcef28831d-istiod-ca-cert\") pod \"0478a844-cbe2-41b0-a4fe-78fcef28831d\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " Oct 31 03:21:31.808139 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808103 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-envoy\") pod \"f25526fb-70e1-44f2-8157-c4648fc91e9a\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " Oct 31 03:21:31.808192 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808176 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-kube-api-access-8frwt\" (UniqueName: \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-kube-api-access-8frwt\") pod \"0478a844-cbe2-41b0-a4fe-78fcef28831d\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " Oct 31 03:21:31.808226 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808209 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-envoy\") pod \"0478a844-cbe2-41b0-a4fe-78fcef28831d\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " Oct 31 03:21:31.808261 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808233 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-data\") pod \"f25526fb-70e1-44f2-8157-c4648fc91e9a\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " Oct 31 03:21:31.808294 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808277 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-kube-api-access-8lzhs\" (UniqueName: \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-kube-api-access-8lzhs\") pod \"f25526fb-70e1-44f2-8157-c4648fc91e9a\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " Oct 31 03:21:31.808337 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808320 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25526fb-70e1-44f2-8157-c4648fc91e9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25526fb-70e1-44f2-8157-c4648fc91e9a-istiod-ca-cert\") pod \"f25526fb-70e1-44f2-8157-c4648fc91e9a\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " Oct 31 03:21:31.808376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808365 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-socket\") pod \"f25526fb-70e1-44f2-8157-c4648fc91e9a\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " Oct 31 03:21:31.808704 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808434 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-certs\") pod \"0478a844-cbe2-41b0-a4fe-78fcef28831d\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " Oct 31 03:21:31.808704 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808470 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-token\" (UniqueName: \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-token\") pod \"0478a844-cbe2-41b0-a4fe-78fcef28831d\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " Oct 31 03:21:31.808704 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808501 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-socket\") pod \"0478a844-cbe2-41b0-a4fe-78fcef28831d\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " Oct 31 03:21:31.808704 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808591 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-credential-socket\") pod \"f25526fb-70e1-44f2-8157-c4648fc91e9a\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " Oct 31 03:21:31.809088 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808717 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-podinfo\") pod \"f25526fb-70e1-44f2-8157-c4648fc91e9a\" (UID: \"f25526fb-70e1-44f2-8157-c4648fc91e9a\") " Oct 31 03:21:31.809088 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.808751 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-data\") pod \"0478a844-cbe2-41b0-a4fe-78fcef28831d\" (UID: \"0478a844-cbe2-41b0-a4fe-78fcef28831d\") " Oct 31 03:21:31.809216 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.809100 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-certs" pod "0478a844-cbe2-41b0-a4fe-78fcef28831d" (UID: "0478a844-cbe2-41b0-a4fe-78fcef28831d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.809216 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.809193 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-certs" pod "f25526fb-70e1-44f2-8157-c4648fc91e9a" (UID: "f25526fb-70e1-44f2-8157-c4648fc91e9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.809355 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.809332 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-data" pod "0478a844-cbe2-41b0-a4fe-78fcef28831d" (UID: "0478a844-cbe2-41b0-a4fe-78fcef28831d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.810610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.809565 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25526fb-70e1-44f2-8157-c4648fc91e9a-istiod-ca-cert" pod "f25526fb-70e1-44f2-8157-c4648fc91e9a" (UID: "f25526fb-70e1-44f2-8157-c4648fc91e9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:31.810610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.809950 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-socket" pod "f25526fb-70e1-44f2-8157-c4648fc91e9a" (UID: "f25526fb-70e1-44f2-8157-c4648fc91e9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.810610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810129 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-credential-socket" pod "0478a844-cbe2-41b0-a4fe-78fcef28831d" (UID: "0478a844-cbe2-41b0-a4fe-78fcef28831d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.810610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810459 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-data" pod "f25526fb-70e1-44f2-8157-c4648fc91e9a" (UID: "f25526fb-70e1-44f2-8157-c4648fc91e9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.810610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810583 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.810610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810606 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810622 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7526b30d-815e-4669-808d-2bf77dbfbc8a-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810637 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25526fb-70e1-44f2-8157-c4648fc91e9a-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810649 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810664 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810677 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810693 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810707 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810721 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ftvf\" (UniqueName: \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-kube-api-access-5ftvf\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810735 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810748 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7526b30d-815e-4669-808d-2bf77dbfbc8a-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810762 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810776 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7526b30d-815e-4669-808d-2bf77dbfbc8a-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810790 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.810788 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0478a844-cbe2-41b0-a4fe-78fcef28831d-istiod-ca-cert" pod "0478a844-cbe2-41b0-a4fe-78fcef28831d" (UID: "0478a844-cbe2-41b0-a4fe-78fcef28831d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:31.811143 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.811073 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-credential-socket" pod "f25526fb-70e1-44f2-8157-c4648fc91e9a" (UID: "f25526fb-70e1-44f2-8157-c4648fc91e9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.811714 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.811261 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-socket" pod "0478a844-cbe2-41b0-a4fe-78fcef28831d" (UID: "0478a844-cbe2-41b0-a4fe-78fcef28831d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.812979 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.812318 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-token" pod "f25526fb-70e1-44f2-8157-c4648fc91e9a" (UID: "f25526fb-70e1-44f2-8157-c4648fc91e9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:31.812979 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.812360 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-podinfo" pod "0478a844-cbe2-41b0-a4fe-78fcef28831d" (UID: "0478a844-cbe2-41b0-a4fe-78fcef28831d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:31.813230 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.813020 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-kube-api-access-8lzhs" pod "f25526fb-70e1-44f2-8157-c4648fc91e9a" (UID: "f25526fb-70e1-44f2-8157-c4648fc91e9a"). InnerVolumeSpecName "kube-api-access-8lzhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:31.813489 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.813464 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-envoy" pod "f25526fb-70e1-44f2-8157-c4648fc91e9a" (UID: "f25526fb-70e1-44f2-8157-c4648fc91e9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.815066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.814408 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-kube-api-access-8frwt" pod "0478a844-cbe2-41b0-a4fe-78fcef28831d" (UID: "0478a844-cbe2-41b0-a4fe-78fcef28831d"). InnerVolumeSpecName "kube-api-access-8frwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:31.815066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.814586 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-envoy" pod "0478a844-cbe2-41b0-a4fe-78fcef28831d" (UID: "0478a844-cbe2-41b0-a4fe-78fcef28831d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:31.815066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.814595 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-podinfo" pod "f25526fb-70e1-44f2-8157-c4648fc91e9a" (UID: "f25526fb-70e1-44f2-8157-c4648fc91e9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:31.815066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.814820 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-token" pod "0478a844-cbe2-41b0-a4fe-78fcef28831d" (UID: "0478a844-cbe2-41b0-a4fe-78fcef28831d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:31.912543 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912509 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0478a844-cbe2-41b0-a4fe-78fcef28831d-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912543 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912535 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912543 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912551 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8frwt\" (UniqueName: \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-kube-api-access-8frwt\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912561 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912570 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912578 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lzhs\" (UniqueName: \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-kube-api-access-8lzhs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912586 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912594 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0478a844-cbe2-41b0-a4fe-78fcef28831d-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912603 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25526fb-70e1-44f2-8157-c4648fc91e9a-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912610 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912618 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0478a844-cbe2-41b0-a4fe-78fcef28831d-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:31.912820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:31.912626 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25526fb-70e1-44f2-8157-c4648fc91e9a-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:32.008719 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.006549 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:32.012443 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.012353 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:32.017258 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.017234 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:21:32.024247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.024205 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6"] Oct 31 03:21:32.030794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.030765 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-qjvk6"] Oct 31 03:21:32.151448 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:21:32.151412 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:21:32.437236 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.437163 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" path="/var/lib/kubelet/pods/7526b30d-815e-4669-808d-2bf77dbfbc8a/volumes" Oct 31 03:21:32.437612 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.437598 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c02eb473-9cd5-4595-926d-e17429e58a8c" path="/var/lib/kubelet/pods/c02eb473-9cd5-4595-926d-e17429e58a8c/volumes" Oct 31 03:21:32.437974 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.437961 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d91df2-e63c-4fd5-ad16-f45a30251fbb" path="/var/lib/kubelet/pods/e0d91df2-e63c-4fd5-ad16-f45a30251fbb/volumes" Oct 31 03:21:32.692855 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.692761 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g" Oct 31 03:21:32.694643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.694563 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" event={"ID":"f25526fb-70e1-44f2-8157-c4648fc91e9a","Type":"ContainerDied","Data":"f5975003b7c614237b2dbecf03bf9026167b6eb7f34d1e14aad5a522ffb8552d"} Oct 31 03:21:32.694772 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.694677 4456 scope.go:122] "RemoveContainer" containerID="8fc96c5e205efbe0f587544fb0508534febab2d7a456edbcbe539b5e1debcdbd" Oct 31 03:21:32.695174 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.695154 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b" Oct 31 03:21:32.726056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.725980 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g"] Oct 31 03:21:32.732791 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.732762 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-4zh8g"] Oct 31 03:21:32.741806 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.741776 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b"] Oct 31 03:21:32.747577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:32.747555 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-pxd6b"] Oct 31 03:21:34.442973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:34.442943 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" path="/var/lib/kubelet/pods/0478a844-cbe2-41b0-a4fe-78fcef28831d/volumes" Oct 31 03:21:34.443968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:34.443950 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" path="/var/lib/kubelet/pods/f25526fb-70e1-44f2-8157-c4648fc91e9a/volumes" Oct 31 03:21:36.553018 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:36.552980 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n"] Oct 31 03:21:36.570460 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:36.570424 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fccff5-rbv6n"] Oct 31 03:21:38.438080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:38.438024 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e96c53-8917-4efa-b766-c9108d3a0d09" path="/var/lib/kubelet/pods/14e96c53-8917-4efa-b766-c9108d3a0d09/volumes" Oct 31 03:21:39.513457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:39.513405 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6"] Oct 31 03:21:39.521608 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:39.521572 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fd2e0b-sbdl6"] Oct 31 03:21:40.437349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:40.437313 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45306927-65b7-4e4f-b86c-b107938ff6f0" path="/var/lib/kubelet/pods/45306927-65b7-4e4f-b86c-b107938ff6f0/volumes" Oct 31 03:21:41.138773 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.138724 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj"] Oct 31 03:21:41.139371 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.138981 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" podUID="90f08352-6eb7-40f5-8672-1de230b9d165" containerName="maas-api" containerID="cri-o://5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f" gracePeriod=30 Oct 31 03:21:41.695055 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.689119 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h"] Oct 31 03:21:41.697760 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.697717 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-99bc97-nrj8h"] Oct 31 03:21:41.799102 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.798341 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:21:41.871060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.864109 4456 generic.go:356] "Generic (PLEG): container finished" podID="90f08352-6eb7-40f5-8672-1de230b9d165" containerID="5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f" exitCode=0 Oct 31 03:21:41.871060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.864422 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" event={"ID":"90f08352-6eb7-40f5-8672-1de230b9d165","Type":"ContainerDied","Data":"5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f"} Oct 31 03:21:41.871060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.864479 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" event={"ID":"90f08352-6eb7-40f5-8672-1de230b9d165","Type":"ContainerDied","Data":"329338c1cd7343155113dfe2ca9d9c5816ce74a3d89bdd13d59974c5cb69a325"} Oct 31 03:21:41.871060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.864512 4456 scope.go:122] "RemoveContainer" containerID="5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f" Oct 31 03:21:41.871060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.864692 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj" Oct 31 03:21:41.881858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.881830 4456 scope.go:122] "RemoveContainer" containerID="5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f" Oct 31 03:21:41.882384 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:21:41.882303 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f\": container with ID starting with 5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f not found: ID does not exist" containerID="5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f" Oct 31 03:21:41.882384 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.882342 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f"} err="failed to get container status \"5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f\": rpc error: code = NotFound desc = could not find container \"5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f\": container with ID starting with 5234476b04e9b0673247ef5afcb1dc71f7acfbcabc50754ef6277878a798814f not found: ID does not exist" Oct 31 03:21:41.941184 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.941152 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-maas-api-tls\") pod \"90f08352-6eb7-40f5-8672-1de230b9d165\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " Oct 31 03:21:41.941325 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.941214 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-metrics-certs\") pod \"90f08352-6eb7-40f5-8672-1de230b9d165\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " Oct 31 03:21:41.941325 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.941266 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f08352-6eb7-40f5-8672-1de230b9d165-kube-api-access-4pgrc\" (UniqueName: \"kubernetes.io/projected/90f08352-6eb7-40f5-8672-1de230b9d165-kube-api-access-4pgrc\") pod \"90f08352-6eb7-40f5-8672-1de230b9d165\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " Oct 31 03:21:41.941420 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.941347 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f08352-6eb7-40f5-8672-1de230b9d165-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f08352-6eb7-40f5-8672-1de230b9d165-openshift-service-ca\") pod \"90f08352-6eb7-40f5-8672-1de230b9d165\" (UID: \"90f08352-6eb7-40f5-8672-1de230b9d165\") " Oct 31 03:21:41.941746 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.941720 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f08352-6eb7-40f5-8672-1de230b9d165-openshift-service-ca" pod "90f08352-6eb7-40f5-8672-1de230b9d165" (UID: "90f08352-6eb7-40f5-8672-1de230b9d165"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:41.941978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.941960 4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f08352-6eb7-40f5-8672-1de230b9d165-openshift-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:41.943434 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.943411 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-maas-api-tls" pod "90f08352-6eb7-40f5-8672-1de230b9d165" (UID: "90f08352-6eb7-40f5-8672-1de230b9d165"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:41.943544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.943445 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-metrics-certs" pod "90f08352-6eb7-40f5-8672-1de230b9d165" (UID: "90f08352-6eb7-40f5-8672-1de230b9d165"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:41.943725 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:41.943704 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f08352-6eb7-40f5-8672-1de230b9d165-kube-api-access-4pgrc" pod "90f08352-6eb7-40f5-8672-1de230b9d165" (UID: "90f08352-6eb7-40f5-8672-1de230b9d165"). InnerVolumeSpecName "kube-api-access-4pgrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:42.043698 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:42.043661 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-maas-api-tls\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:42.043698 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:42.043692 4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90f08352-6eb7-40f5-8672-1de230b9d165-metrics-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:42.043698 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:42.043703 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pgrc\" (UniqueName: \"kubernetes.io/projected/90f08352-6eb7-40f5-8672-1de230b9d165-kube-api-access-4pgrc\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:42.260869 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:42.260830 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj"] Oct 31 03:21:42.282000 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:42.281956 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-99bc97-84f57f6b79-cxnqj"] Oct 31 03:21:42.436788 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:42.436707 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb55fff-c668-4b41-902b-01c0a86b0e63" path="/var/lib/kubelet/pods/0cb55fff-c668-4b41-902b-01c0a86b0e63/volumes" Oct 31 03:21:42.437384 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:42.437367 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f08352-6eb7-40f5-8672-1de230b9d165" path="/var/lib/kubelet/pods/90f08352-6eb7-40f5-8672-1de230b9d165/volumes" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.843866 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw"] Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845162 4456 cpu_manager.go:414] "containerMap: removing container" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845185 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845213 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b4cffa9e-6a33-458c-b16c-203213456d12" containerName="revoke-keys" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845222 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4cffa9e-6a33-458c-b16c-203213456d12" containerName="revoke-keys" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845236 4456 cpu_manager.go:414] "containerMap: removing container" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845244 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845262 4456 cpu_manager.go:414] "containerMap: removing container" podUID="c02eb473-9cd5-4595-926d-e17429e58a8c" containerName="payload-pre-processing" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845270 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02eb473-9cd5-4595-926d-e17429e58a8c" containerName="payload-pre-processing" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845286 4456 cpu_manager.go:414] "containerMap: removing container" podUID="e0d91df2-e63c-4fd5-ad16-f45a30251fbb" containerName="payload-processing" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845295 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d91df2-e63c-4fd5-ad16-f45a30251fbb" containerName="payload-processing" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845331 4456 cpu_manager.go:414] "containerMap: removing container" podUID="e4bc60b4-98d7-4005-97ea-1cadc44f7aab" containerName="revoke-keys" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845338 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bc60b4-98d7-4005-97ea-1cadc44f7aab" containerName="revoke-keys" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845395 4456 cpu_manager.go:414] "containerMap: removing container" podUID="90f08352-6eb7-40f5-8672-1de230b9d165" containerName="maas-api" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845404 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f08352-6eb7-40f5-8672-1de230b9d165" containerName="maas-api" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845419 4456 cpu_manager.go:414] "containerMap: removing container" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845427 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845658 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4cffa9e-6a33-458c-b16c-203213456d12" containerName="revoke-keys" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845676 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25526fb-70e1-44f2-8157-c4648fc91e9a" containerName="istio-proxy" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845686 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02eb473-9cd5-4595-926d-e17429e58a8c" containerName="payload-pre-processing" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845697 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d91df2-e63c-4fd5-ad16-f45a30251fbb" containerName="payload-processing" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845710 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bc60b4-98d7-4005-97ea-1cadc44f7aab" containerName="revoke-keys" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845728 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f08352-6eb7-40f5-8672-1de230b9d165" containerName="maas-api" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845739 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0478a844-cbe2-41b0-a4fe-78fcef28831d" containerName="istio-proxy" Oct 31 03:21:46.846082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.845751 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7526b30d-815e-4669-808d-2bf77dbfbc8a" containerName="istio-proxy" Oct 31 03:21:46.873652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.868101 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw"] Oct 31 03:21:46.873652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.868183 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:46.876706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.876685 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:21:46.876873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.876861 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:21:46.986473 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.986440 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:46.986706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.986661 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5dx8\" (UniqueName: \"kubernetes.io/projected/934b7bb6-89b4-4cd3-b746-037be2725481-kube-api-access-d5dx8\") pod \"maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:46.986794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:46.986709 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:47.088640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:47.088603 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5dx8\" (UniqueName: \"kubernetes.io/projected/934b7bb6-89b4-4cd3-b746-037be2725481-kube-api-access-d5dx8\") pod \"maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:47.088829 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:47.088655 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:47.088912 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:47.088826 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:47.089538 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:47.089509 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:47.091129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:47.091108 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:47.107014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:47.106930 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5dx8\" (UniqueName: \"kubernetes.io/projected/934b7bb6-89b4-4cd3-b746-037be2725481-kube-api-access-d5dx8\") pod \"maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:47.189790 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:47.189754 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:47.486380 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:21:47.486336 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934b7bb6_89b4_4cd3_b746_037be2725481.slice/crio-58057879cd298507fe8adc55e337cefc8b4c9298069815b45cfaaa086c324cc7 WatchSource:0}: Error finding container 58057879cd298507fe8adc55e337cefc8b4c9298069815b45cfaaa086c324cc7: Status 404 returned error can't find the container with id 58057879cd298507fe8adc55e337cefc8b4c9298069815b45cfaaa086c324cc7 Oct 31 03:21:47.488100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:47.488031 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw"] Oct 31 03:21:48.094695 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:48.094662 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" event={"ID":"934b7bb6-89b4-4cd3-b746-037be2725481","Type":"ContainerStarted","Data":"080a7cde0fea8031e8a42cab4b5a57a4069798dcf54bce2f1aa826643f87d4c4"} Oct 31 03:21:48.095151 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:48.094713 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" event={"ID":"934b7bb6-89b4-4cd3-b746-037be2725481","Type":"ContainerStarted","Data":"58057879cd298507fe8adc55e337cefc8b4c9298069815b45cfaaa086c324cc7"} Oct 31 03:21:48.138494 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:48.138407 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" podStartSLOduration=2.138393574 podStartE2EDuration="2.138393574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:48.130981914 +0000 UTC m=+1589.811508042" watchObservedRunningTime="2026-10-31 03:21:48.138393574 +0000 UTC m=+1589.818919701" Oct 31 03:21:49.108547 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.108512 4456 generic.go:356] "Generic (PLEG): container finished" podID="934b7bb6-89b4-4cd3-b746-037be2725481" containerID="080a7cde0fea8031e8a42cab4b5a57a4069798dcf54bce2f1aa826643f87d4c4" exitCode=0 Oct 31 03:21:49.109075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.108591 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" event={"ID":"934b7bb6-89b4-4cd3-b746-037be2725481","Type":"ContainerDied","Data":"080a7cde0fea8031e8a42cab4b5a57a4069798dcf54bce2f1aa826643f87d4c4"} Oct 31 03:21:49.794163 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.794129 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6"] Oct 31 03:21:49.798972 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.798949 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:49.821075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.816896 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6"] Oct 31 03:21:49.869145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.868912 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:49.869347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.869180 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfqvw\" (UniqueName: \"kubernetes.io/projected/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-kube-api-access-vfqvw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:49.869347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.869247 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:49.970485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.970441 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:49.970679 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.970666 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfqvw\" (UniqueName: \"kubernetes.io/projected/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-kube-api-access-vfqvw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:49.970750 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.970722 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:49.971494 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.971464 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:49.973136 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.973106 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:49.988677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:49.988642 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfqvw\" (UniqueName: \"kubernetes.io/projected/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-kube-api-access-vfqvw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:50.167724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.167689 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:50.345968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.345942 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:50.380925 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.380841 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-openshift-ca\") pod \"934b7bb6-89b4-4cd3-b746-037be2725481\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " Oct 31 03:21:50.380925 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.380899 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934b7bb6-89b4-4cd3-b746-037be2725481-kube-api-access-d5dx8\" (UniqueName: \"kubernetes.io/projected/934b7bb6-89b4-4cd3-b746-037be2725481-kube-api-access-d5dx8\") pod \"934b7bb6-89b4-4cd3-b746-037be2725481\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " Oct 31 03:21:50.381225 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.380940 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-xks-ca\") pod \"934b7bb6-89b4-4cd3-b746-037be2725481\" (UID: \"934b7bb6-89b4-4cd3-b746-037be2725481\") " Oct 31 03:21:50.382063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.382005 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-openshift-ca" pod "934b7bb6-89b4-4cd3-b746-037be2725481" (UID: "934b7bb6-89b4-4cd3-b746-037be2725481"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:50.386141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.386108 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934b7bb6-89b4-4cd3-b746-037be2725481-kube-api-access-d5dx8" pod "934b7bb6-89b4-4cd3-b746-037be2725481" (UID: "934b7bb6-89b4-4cd3-b746-037be2725481"). InnerVolumeSpecName "kube-api-access-d5dx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:50.386296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.386274 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-xks-ca" pod "934b7bb6-89b4-4cd3-b746-037be2725481" (UID: "934b7bb6-89b4-4cd3-b746-037be2725481"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:50.485890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.485846 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:50.485890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.485881 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5dx8\" (UniqueName: \"kubernetes.io/projected/934b7bb6-89b4-4cd3-b746-037be2725481-kube-api-access-d5dx8\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:50.486175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.485900 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/934b7bb6-89b4-4cd3-b746-037be2725481-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:50.517679 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:50.517226 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6"] Oct 31 03:21:51.132224 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:51.132149 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" Oct 31 03:21:51.132224 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:51.132186 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw" event={"ID":"934b7bb6-89b4-4cd3-b746-037be2725481","Type":"ContainerDied","Data":"58057879cd298507fe8adc55e337cefc8b4c9298069815b45cfaaa086c324cc7"} Oct 31 03:21:51.132439 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:51.132232 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58057879cd298507fe8adc55e337cefc8b4c9298069815b45cfaaa086c324cc7" Oct 31 03:21:51.134253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:51.134225 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" event={"ID":"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d","Type":"ContainerStarted","Data":"a92d97962f83f9ea1f3e05575eb40c6df637f82a64a5b266bafada66a71e75d3"} Oct 31 03:21:51.134409 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:51.134275 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" event={"ID":"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d","Type":"ContainerStarted","Data":"0fcb471db726ccb1204c163c0dbe99f5f88dc1024f4f907508e83ffe6a4a1ef0"} Oct 31 03:21:51.161965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:51.161901 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" podStartSLOduration=2.161883361 podStartE2EDuration="2.161883361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:51.156938912 +0000 UTC m=+1592.837465039" watchObservedRunningTime="2026-10-31 03:21:51.161883361 +0000 UTC m=+1592.842409493" Oct 31 03:21:52.176059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:52.175953 4456 generic.go:356] "Generic (PLEG): container finished" podID="37c79a8e-59ee-42e6-a4dc-f065a28a7a2d" containerID="a92d97962f83f9ea1f3e05575eb40c6df637f82a64a5b266bafada66a71e75d3" exitCode=0 Oct 31 03:21:52.176662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:52.176119 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" event={"ID":"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d","Type":"ContainerDied","Data":"a92d97962f83f9ea1f3e05575eb40c6df637f82a64a5b266bafada66a71e75d3"} Oct 31 03:21:53.499742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.499702 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:53.576181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.576095 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-openshift-ca\") pod \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " Oct 31 03:21:53.576476 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.576451 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-kube-api-access-vfqvw\" (UniqueName: \"kubernetes.io/projected/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-kube-api-access-vfqvw\") pod \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " Oct 31 03:21:53.576711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.576690 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-xks-ca\") pod \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\" (UID: \"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d\") " Oct 31 03:21:53.579437 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.579408 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-openshift-ca" pod "37c79a8e-59ee-42e6-a4dc-f065a28a7a2d" (UID: "37c79a8e-59ee-42e6-a4dc-f065a28a7a2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:53.582281 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.582254 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-kube-api-access-vfqvw" pod "37c79a8e-59ee-42e6-a4dc-f065a28a7a2d" (UID: "37c79a8e-59ee-42e6-a4dc-f065a28a7a2d"). InnerVolumeSpecName "kube-api-access-vfqvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:53.593886 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.593833 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-xks-ca" pod "37c79a8e-59ee-42e6-a4dc-f065a28a7a2d" (UID: "37c79a8e-59ee-42e6-a4dc-f065a28a7a2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:53.678522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.678479 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:53.678522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.678513 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfqvw\" (UniqueName: \"kubernetes.io/projected/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-kube-api-access-vfqvw\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:53.678522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.678524 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:53.770192 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.768440 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv"] Oct 31 03:21:53.770192 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.768760 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" containerID="cri-o://bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085" gracePeriod=30 Oct 31 03:21:53.791697 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.788729 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g"] Oct 31 03:21:53.791697 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.789792 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" containerID="cri-o://7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9" gracePeriod=30 Oct 31 03:21:53.810297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.809574 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl"] Oct 31 03:21:53.810297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:53.809825 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" containerID="cri-o://594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78" gracePeriod=30 Oct 31 03:21:54.265859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:54.265831 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" Oct 31 03:21:54.266075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:54.265864 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6" event={"ID":"37c79a8e-59ee-42e6-a4dc-f065a28a7a2d","Type":"ContainerDied","Data":"0fcb471db726ccb1204c163c0dbe99f5f88dc1024f4f907508e83ffe6a4a1ef0"} Oct 31 03:21:54.266075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:54.265914 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fcb471db726ccb1204c163c0dbe99f5f88dc1024f4f907508e83ffe6a4a1ef0" Oct 31 03:21:56.114045 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:56.113989 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:21:56.114436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:56.114086 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:21:56.887592 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:56.887550 4456 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:21:56.887802 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:56.887626 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:21:57.004786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:57.004749 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5"] Oct 31 03:21:57.025987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:57.025941 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z"] Oct 31 03:21:59.182056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.159888 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:21:59.182056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.171639 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:21:59.208242 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208206 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-kube-api-access-tb4ch\" (UniqueName: \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-kube-api-access-tb4ch\") pod \"4657dd4e-e0de-4400-8304-74c7794421f5\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " Oct 31 03:21:59.208427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208293 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-istio-token\" (UniqueName: \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-istio-token\") pod \"4657dd4e-e0de-4400-8304-74c7794421f5\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " Oct 31 03:21:59.208427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208389 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-kube-api-access-7xp6r\" (UniqueName: \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-kube-api-access-7xp6r\") pod \"7d087ee5-7714-4cc3-88a8-de6e13297913\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " Oct 31 03:21:59.208565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208426 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-envoy\") pod \"4657dd4e-e0de-4400-8304-74c7794421f5\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " Oct 31 03:21:59.208565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208485 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4657dd4e-e0de-4400-8304-74c7794421f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4657dd4e-e0de-4400-8304-74c7794421f5-istio-podinfo\") pod \"4657dd4e-e0de-4400-8304-74c7794421f5\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " Oct 31 03:21:59.208676 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208568 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-envoy\") pod \"7d087ee5-7714-4cc3-88a8-de6e13297913\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " Oct 31 03:21:59.208676 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208632 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-credential-socket\") pod \"4657dd4e-e0de-4400-8304-74c7794421f5\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " Oct 31 03:21:59.208799 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208720 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-podinfo\") pod \"7d087ee5-7714-4cc3-88a8-de6e13297913\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " Oct 31 03:21:59.208864 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208815 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-certs\") pod \"7d087ee5-7714-4cc3-88a8-de6e13297913\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " Oct 31 03:21:59.208923 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208875 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-data\") pod \"4657dd4e-e0de-4400-8304-74c7794421f5\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " Oct 31 03:21:59.208982 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208953 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-socket\") pod \"4657dd4e-e0de-4400-8304-74c7794421f5\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " Oct 31 03:21:59.209058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.208980 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-credential-socket\") pod \"7d087ee5-7714-4cc3-88a8-de6e13297913\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " Oct 31 03:21:59.209058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.209025 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-certs\") pod \"4657dd4e-e0de-4400-8304-74c7794421f5\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " Oct 31 03:21:59.209178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.209083 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-token\" (UniqueName: \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-token\") pod \"7d087ee5-7714-4cc3-88a8-de6e13297913\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " Oct 31 03:21:59.209178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.209161 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4657dd4e-e0de-4400-8304-74c7794421f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4657dd4e-e0de-4400-8304-74c7794421f5-istiod-ca-cert\") pod \"4657dd4e-e0de-4400-8304-74c7794421f5\" (UID: \"4657dd4e-e0de-4400-8304-74c7794421f5\") " Oct 31 03:21:59.209283 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.209243 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d087ee5-7714-4cc3-88a8-de6e13297913-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d087ee5-7714-4cc3-88a8-de6e13297913-istiod-ca-cert\") pod \"7d087ee5-7714-4cc3-88a8-de6e13297913\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " Oct 31 03:21:59.209283 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.209271 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-socket\") pod \"7d087ee5-7714-4cc3-88a8-de6e13297913\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " Oct 31 03:21:59.209397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.209301 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-data\") pod \"7d087ee5-7714-4cc3-88a8-de6e13297913\" (UID: \"7d087ee5-7714-4cc3-88a8-de6e13297913\") " Oct 31 03:21:59.210504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.210172 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-data" pod "7d087ee5-7714-4cc3-88a8-de6e13297913" (UID: "7d087ee5-7714-4cc3-88a8-de6e13297913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.210504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.210477 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-data" pod "4657dd4e-e0de-4400-8304-74c7794421f5" (UID: "4657dd4e-e0de-4400-8304-74c7794421f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.211020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.210992 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4657dd4e-e0de-4400-8304-74c7794421f5-istiod-ca-cert" pod "4657dd4e-e0de-4400-8304-74c7794421f5" (UID: "4657dd4e-e0de-4400-8304-74c7794421f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:59.211180 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.211073 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d087ee5-7714-4cc3-88a8-de6e13297913-istiod-ca-cert" pod "7d087ee5-7714-4cc3-88a8-de6e13297913" (UID: "7d087ee5-7714-4cc3-88a8-de6e13297913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:59.211244 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.211216 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-socket" pod "7d087ee5-7714-4cc3-88a8-de6e13297913" (UID: "7d087ee5-7714-4cc3-88a8-de6e13297913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.211445 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.211421 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-credential-socket" pod "7d087ee5-7714-4cc3-88a8-de6e13297913" (UID: "7d087ee5-7714-4cc3-88a8-de6e13297913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.211517 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.211428 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-certs" pod "4657dd4e-e0de-4400-8304-74c7794421f5" (UID: "4657dd4e-e0de-4400-8304-74c7794421f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.211643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.211624 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-credential-socket" pod "4657dd4e-e0de-4400-8304-74c7794421f5" (UID: "4657dd4e-e0de-4400-8304-74c7794421f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.211711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.211640 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-socket" pod "4657dd4e-e0de-4400-8304-74c7794421f5" (UID: "4657dd4e-e0de-4400-8304-74c7794421f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.212193 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.212167 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-certs" pod "7d087ee5-7714-4cc3-88a8-de6e13297913" (UID: "7d087ee5-7714-4cc3-88a8-de6e13297913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.214836 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.214806 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-token" pod "7d087ee5-7714-4cc3-88a8-de6e13297913" (UID: "7d087ee5-7714-4cc3-88a8-de6e13297913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.214927 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.214850 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-istio-token" pod "4657dd4e-e0de-4400-8304-74c7794421f5" (UID: "4657dd4e-e0de-4400-8304-74c7794421f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.214927 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.214874 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-podinfo" pod "7d087ee5-7714-4cc3-88a8-de6e13297913" (UID: "7d087ee5-7714-4cc3-88a8-de6e13297913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:59.215631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.215602 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-kube-api-access-tb4ch" pod "4657dd4e-e0de-4400-8304-74c7794421f5" (UID: "4657dd4e-e0de-4400-8304-74c7794421f5"). InnerVolumeSpecName "kube-api-access-tb4ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.216423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.216392 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4657dd4e-e0de-4400-8304-74c7794421f5-istio-podinfo" pod "4657dd4e-e0de-4400-8304-74c7794421f5" (UID: "4657dd4e-e0de-4400-8304-74c7794421f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:59.216778 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.216754 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-envoy" pod "4657dd4e-e0de-4400-8304-74c7794421f5" (UID: "4657dd4e-e0de-4400-8304-74c7794421f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.224591 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.224565 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-kube-api-access-7xp6r" pod "7d087ee5-7714-4cc3-88a8-de6e13297913" (UID: "7d087ee5-7714-4cc3-88a8-de6e13297913"). InnerVolumeSpecName "kube-api-access-7xp6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.224703 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.224610 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-envoy" pod "7d087ee5-7714-4cc3-88a8-de6e13297913" (UID: "7d087ee5-7714-4cc3-88a8-de6e13297913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.242973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.242941 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:21:59.310313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.310270 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-envoy\") pod \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " Oct 31 03:21:59.310487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.310370 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-token\" (UniqueName: \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-token\") pod \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " Oct 31 03:21:59.310487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.310433 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istiod-ca-cert\") pod \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " Oct 31 03:21:59.310487 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.310474 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-data\") pod \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " Oct 31 03:21:59.310687 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.310513 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-podinfo\") pod \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " Oct 31 03:21:59.310687 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.310621 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-kube-api-access-976fq\" (UniqueName: \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-kube-api-access-976fq\") pod \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " Oct 31 03:21:59.310795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.310698 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-socket\") pod \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " Oct 31 03:21:59.310795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.310729 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-certs\") pod \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " Oct 31 03:21:59.310907 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.310812 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-credential-socket\") pod \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\" (UID: \"cfc17f71-5f77-48e0-a0c9-395bb51b4388\") " Oct 31 03:21:59.310964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.310907 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istiod-ca-cert" pod "cfc17f71-5f77-48e0-a0c9-395bb51b4388" (UID: "cfc17f71-5f77-48e0-a0c9-395bb51b4388"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:59.311264 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.311171 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-data" pod "cfc17f71-5f77-48e0-a0c9-395bb51b4388" (UID: "cfc17f71-5f77-48e0-a0c9-395bb51b4388"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.311382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.311301 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-socket" pod "cfc17f71-5f77-48e0-a0c9-395bb51b4388" (UID: "cfc17f71-5f77-48e0-a0c9-395bb51b4388"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.311457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.311381 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-credential-socket" pod "cfc17f71-5f77-48e0-a0c9-395bb51b4388" (UID: "cfc17f71-5f77-48e0-a0c9-395bb51b4388"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.311612 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.311574 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-certs" pod "cfc17f71-5f77-48e0-a0c9-395bb51b4388" (UID: "cfc17f71-5f77-48e0-a0c9-395bb51b4388"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.313209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313177 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-envoy" pod "cfc17f71-5f77-48e0-a0c9-395bb51b4388" (UID: "cfc17f71-5f77-48e0-a0c9-395bb51b4388"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.313481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313451 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-kube-api-access-976fq" pod "cfc17f71-5f77-48e0-a0c9-395bb51b4388" (UID: "cfc17f71-5f77-48e0-a0c9-395bb51b4388"). InnerVolumeSpecName "kube-api-access-976fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.313481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313477 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313496 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313512 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313525 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313538 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313553 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313569 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4657dd4e-e0de-4400-8304-74c7794421f5-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313583 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313596 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d087ee5-7714-4cc3-88a8-de6e13297913-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313609 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313625 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313637 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313650 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb4ch\" (UniqueName: \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-kube-api-access-tb4ch\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.313666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313664 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4657dd4e-e0de-4400-8304-74c7794421f5-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313678 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xp6r\" (UniqueName: \"kubernetes.io/projected/7d087ee5-7714-4cc3-88a8-de6e13297913-kube-api-access-7xp6r\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313691 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313704 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4657dd4e-e0de-4400-8304-74c7794421f5-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313718 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313731 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313745 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313758 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313773 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d087ee5-7714-4cc3-88a8-de6e13297913-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313787 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d087ee5-7714-4cc3-88a8-de6e13297913-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314187 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.313801 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4657dd4e-e0de-4400-8304-74c7794421f5-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.314568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.314542 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-podinfo" pod "cfc17f71-5f77-48e0-a0c9-395bb51b4388" (UID: "cfc17f71-5f77-48e0-a0c9-395bb51b4388"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:59.315240 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.315217 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-token" pod "cfc17f71-5f77-48e0-a0c9-395bb51b4388" (UID: "cfc17f71-5f77-48e0-a0c9-395bb51b4388"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.356138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.355743 4456 generic.go:356] "Generic (PLEG): container finished" podID="4657dd4e-e0de-4400-8304-74c7794421f5" containerID="7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9" exitCode=0 Oct 31 03:21:59.356138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.355830 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" event={"ID":"4657dd4e-e0de-4400-8304-74c7794421f5","Type":"ContainerDied","Data":"7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9"} Oct 31 03:21:59.356138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.355883 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" event={"ID":"4657dd4e-e0de-4400-8304-74c7794421f5","Type":"ContainerDied","Data":"cb1a972a226b129b3593b2f426260572aa8ce61e9e63d1ca51290712770218a5"} Oct 31 03:21:59.356138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.355920 4456 scope.go:122] "RemoveContainer" containerID="7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9" Oct 31 03:21:59.356138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.355967 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g" Oct 31 03:21:59.362298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.360410 4456 generic.go:356] "Generic (PLEG): container finished" podID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerID="bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085" exitCode=0 Oct 31 03:21:59.362298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.360438 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" event={"ID":"7d087ee5-7714-4cc3-88a8-de6e13297913","Type":"ContainerDied","Data":"bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085"} Oct 31 03:21:59.362298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.360466 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" Oct 31 03:21:59.362298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.360491 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv" event={"ID":"7d087ee5-7714-4cc3-88a8-de6e13297913","Type":"ContainerDied","Data":"d7b2d290acd75f87632518d71155e48c58ea0cec4d1f9b270bc9bb1601f36f36"} Oct 31 03:21:59.363715 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.363689 4456 generic.go:356] "Generic (PLEG): container finished" podID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerID="594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78" exitCode=0 Oct 31 03:21:59.363997 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.363979 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" event={"ID":"cfc17f71-5f77-48e0-a0c9-395bb51b4388","Type":"ContainerDied","Data":"594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78"} Oct 31 03:21:59.364184 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.364021 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" event={"ID":"cfc17f71-5f77-48e0-a0c9-395bb51b4388","Type":"ContainerDied","Data":"3095badd1b5504057ad7434b5e1bbbf4698595fdc1b532a94e853c76fd5d02b9"} Oct 31 03:21:59.364184 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.364109 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl" Oct 31 03:21:59.403774 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.403751 4456 scope.go:122] "RemoveContainer" containerID="7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9" Oct 31 03:21:59.404096 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:21:59.404070 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9\": container with ID starting with 7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9 not found: ID does not exist" containerID="7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9" Oct 31 03:21:59.404218 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.404103 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9"} err="failed to get container status \"7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9\": rpc error: code = NotFound desc = could not find container \"7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9\": container with ID starting with 7771d45c35c41f2845acdf08fe162dc090f3209c48f1bf514303c0d84833f1a9 not found: ID does not exist" Oct 31 03:21:59.404218 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.404119 4456 scope.go:122] "RemoveContainer" containerID="bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085" Oct 31 03:21:59.415715 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.415661 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.415715 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.415698 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-976fq\" (UniqueName: \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-kube-api-access-976fq\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.415715 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.415715 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc17f71-5f77-48e0-a0c9-395bb51b4388-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.416990 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.416959 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g"] Oct 31 03:21:59.418737 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.418713 4456 scope.go:122] "RemoveContainer" containerID="bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085" Oct 31 03:21:59.419084 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:21:59.419058 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085\": container with ID starting with bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085 not found: ID does not exist" containerID="bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085" Oct 31 03:21:59.419190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.419097 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085"} err="failed to get container status \"bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085\": rpc error: code = NotFound desc = could not find container \"bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085\": container with ID starting with bd6279932c8cbab233860cd29f37efb6bc557278fe998438946e3c6e3c689085 not found: ID does not exist" Oct 31 03:21:59.419190 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.419117 4456 scope.go:122] "RemoveContainer" containerID="594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78" Oct 31 03:21:59.422598 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.422566 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-j284g"] Oct 31 03:21:59.430073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.430053 4456 scope.go:122] "RemoveContainer" containerID="594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78" Oct 31 03:21:59.430336 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:21:59.430315 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78\": container with ID starting with 594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78 not found: ID does not exist" containerID="594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78" Oct 31 03:21:59.430396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.430347 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78"} err="failed to get container status \"594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78\": rpc error: code = NotFound desc = could not find container \"594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78\": container with ID starting with 594995055de1015bd4a382462dcd5da3e71bc99115eaae92d20affb474d34b78 not found: ID does not exist" Oct 31 03:21:59.437702 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.437679 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv"] Oct 31 03:21:59.442128 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.442108 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-8nqdv"] Oct 31 03:21:59.468079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.463616 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl"] Oct 31 03:21:59.470542 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.470515 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-dw4tl"] Oct 31 03:21:59.900681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:21:59.900640 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg"] Oct 31 03:22:00.441479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:00.441446 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" path="/var/lib/kubelet/pods/4657dd4e-e0de-4400-8304-74c7794421f5/volumes" Oct 31 03:22:00.442140 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:00.442119 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" path="/var/lib/kubelet/pods/7d087ee5-7714-4cc3-88a8-de6e13297913/volumes" Oct 31 03:22:00.442650 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:00.442633 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" path="/var/lib/kubelet/pods/cfc17f71-5f77-48e0-a0c9-395bb51b4388/volumes" Oct 31 03:22:02.005203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.005160 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" podUID="2dcf78bf-69a6-42b8-99c9-ee5f51323973" containerName="payload-processing" containerID="cri-o://960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de" gracePeriod=25 Oct 31 03:22:02.027267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.027234 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" podUID="186c3940-33eb-42f6-93e2-def354123519" containerName="payload-pre-processing" containerID="cri-o://dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c" gracePeriod=25 Oct 31 03:22:02.137816 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:22:02.137779 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:22:02.357297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.357269 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:22:02.361344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.361317 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:22:02.406589 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.406552 4456 generic.go:356] "Generic (PLEG): container finished" podID="186c3940-33eb-42f6-93e2-def354123519" containerID="dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c" exitCode=0 Oct 31 03:22:02.406765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.406602 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" Oct 31 03:22:02.406765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.406639 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" event={"ID":"186c3940-33eb-42f6-93e2-def354123519","Type":"ContainerDied","Data":"dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c"} Oct 31 03:22:02.406765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.406713 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z" event={"ID":"186c3940-33eb-42f6-93e2-def354123519","Type":"ContainerDied","Data":"5a34d61b1965019a8b88499aef8a7b75f2e0eabefd1d9d5a55a9064a95f41727"} Oct 31 03:22:02.406765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.406745 4456 scope.go:122] "RemoveContainer" containerID="dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c" Oct 31 03:22:02.408904 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.408877 4456 generic.go:356] "Generic (PLEG): container finished" podID="2dcf78bf-69a6-42b8-99c9-ee5f51323973" containerID="960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de" exitCode=0 Oct 31 03:22:02.409062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.408929 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" Oct 31 03:22:02.409062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.408958 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" event={"ID":"2dcf78bf-69a6-42b8-99c9-ee5f51323973","Type":"ContainerDied","Data":"960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de"} Oct 31 03:22:02.409062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.409013 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5" event={"ID":"2dcf78bf-69a6-42b8-99c9-ee5f51323973","Type":"ContainerDied","Data":"5bd5a87789ea8eb94e1caea226e30792a2303898241a4e83a255651a47a7f36a"} Oct 31 03:22:02.419005 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.418983 4456 scope.go:122] "RemoveContainer" containerID="dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c" Oct 31 03:22:02.419529 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:22:02.419504 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c\": container with ID starting with dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c not found: ID does not exist" containerID="dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c" Oct 31 03:22:02.419653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.419540 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c"} err="failed to get container status \"dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c\": rpc error: code = NotFound desc = could not find container \"dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c\": container with ID starting with dfc3ea9cb9703e4ba6b9d44a0c6c5bdf18f605753df422adec9740c29b681c9c not found: ID does not exist" Oct 31 03:22:02.419653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.419566 4456 scope.go:122] "RemoveContainer" containerID="960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de" Oct 31 03:22:02.433460 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.433432 4456 scope.go:122] "RemoveContainer" containerID="960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de" Oct 31 03:22:02.433831 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:22:02.433800 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de\": container with ID starting with 960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de not found: ID does not exist" containerID="960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de" Oct 31 03:22:02.433957 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.433873 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de"} err="failed to get container status \"960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de\": rpc error: code = NotFound desc = could not find container \"960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de\": container with ID starting with 960851e30488df6d62213a4fe3e5778e0c567149ba91a4f46e35f94d53f4f9de not found: ID does not exist" Oct 31 03:22:02.464981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.464945 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186c3940-33eb-42f6-93e2-def354123519-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186c3940-33eb-42f6-93e2-def354123519-plugins-config-volume\") pod \"186c3940-33eb-42f6-93e2-def354123519\" (UID: \"186c3940-33eb-42f6-93e2-def354123519\") " Oct 31 03:22:02.465332 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.465061 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcf78bf-69a6-42b8-99c9-ee5f51323973-kube-api-access-cdlr4\" (UniqueName: \"kubernetes.io/projected/2dcf78bf-69a6-42b8-99c9-ee5f51323973-kube-api-access-cdlr4\") pod \"2dcf78bf-69a6-42b8-99c9-ee5f51323973\" (UID: \"2dcf78bf-69a6-42b8-99c9-ee5f51323973\") " Oct 31 03:22:02.465332 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.465135 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcf78bf-69a6-42b8-99c9-ee5f51323973-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dcf78bf-69a6-42b8-99c9-ee5f51323973-plugins-config-volume\") pod \"2dcf78bf-69a6-42b8-99c9-ee5f51323973\" (UID: \"2dcf78bf-69a6-42b8-99c9-ee5f51323973\") " Oct 31 03:22:02.465332 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.465246 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186c3940-33eb-42f6-93e2-def354123519-kube-api-access-dh8zs\" (UniqueName: \"kubernetes.io/projected/186c3940-33eb-42f6-93e2-def354123519-kube-api-access-dh8zs\") pod \"186c3940-33eb-42f6-93e2-def354123519\" (UID: \"186c3940-33eb-42f6-93e2-def354123519\") " Oct 31 03:22:02.466640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.466616 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186c3940-33eb-42f6-93e2-def354123519-plugins-config-volume" pod "186c3940-33eb-42f6-93e2-def354123519" (UID: "186c3940-33eb-42f6-93e2-def354123519"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:02.466959 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.466932 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcf78bf-69a6-42b8-99c9-ee5f51323973-plugins-config-volume" pod "2dcf78bf-69a6-42b8-99c9-ee5f51323973" (UID: "2dcf78bf-69a6-42b8-99c9-ee5f51323973"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:02.468973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.468945 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcf78bf-69a6-42b8-99c9-ee5f51323973-kube-api-access-cdlr4" pod "2dcf78bf-69a6-42b8-99c9-ee5f51323973" (UID: "2dcf78bf-69a6-42b8-99c9-ee5f51323973"). InnerVolumeSpecName "kube-api-access-cdlr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:02.469629 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.469596 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186c3940-33eb-42f6-93e2-def354123519-kube-api-access-dh8zs" pod "186c3940-33eb-42f6-93e2-def354123519" (UID: "186c3940-33eb-42f6-93e2-def354123519"). InnerVolumeSpecName "kube-api-access-dh8zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:02.567588 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.567484 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh8zs\" (UniqueName: \"kubernetes.io/projected/186c3940-33eb-42f6-93e2-def354123519-kube-api-access-dh8zs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:02.567588 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.567532 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186c3940-33eb-42f6-93e2-def354123519-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:02.567588 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.567543 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdlr4\" (UniqueName: \"kubernetes.io/projected/2dcf78bf-69a6-42b8-99c9-ee5f51323973-kube-api-access-cdlr4\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:02.567588 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.567553 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dcf78bf-69a6-42b8-99c9-ee5f51323973-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:02.769659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.769093 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5"] Oct 31 03:22:02.776780 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.776730 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbbc81-5fc6b64549-gf5s5"] Oct 31 03:22:02.782488 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.782456 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z"] Oct 31 03:22:02.787508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:02.787483 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbbc81-68d6fbc987-kvz8z"] Oct 31 03:22:04.413023 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:04.412972 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6"] Oct 31 03:22:04.419689 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:04.419664 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-677281-9vfz6"] Oct 31 03:22:04.444220 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:04.444187 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186c3940-33eb-42f6-93e2-def354123519" path="/var/lib/kubelet/pods/186c3940-33eb-42f6-93e2-def354123519/volumes" Oct 31 03:22:04.444681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:04.444667 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dcf78bf-69a6-42b8-99c9-ee5f51323973" path="/var/lib/kubelet/pods/2dcf78bf-69a6-42b8-99c9-ee5f51323973/volumes" Oct 31 03:22:04.445102 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:04.445088 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c79a8e-59ee-42e6-a4dc-f065a28a7a2d" path="/var/lib/kubelet/pods/37c79a8e-59ee-42e6-a4dc-f065a28a7a2d/volumes" Oct 31 03:22:04.901515 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:04.901418 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" podUID="30e07052-2f47-420f-a107-c9b3c51b210d" containerName="payload-processing" containerID="cri-o://bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce" gracePeriod=25 Oct 31 03:22:05.207340 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.207310 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:22:05.326550 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.326518 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e07052-2f47-420f-a107-c9b3c51b210d-kube-api-access-mmkgk\" (UniqueName: \"kubernetes.io/projected/30e07052-2f47-420f-a107-c9b3c51b210d-kube-api-access-mmkgk\") pod \"30e07052-2f47-420f-a107-c9b3c51b210d\" (UID: \"30e07052-2f47-420f-a107-c9b3c51b210d\") " Oct 31 03:22:05.326731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.326561 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e07052-2f47-420f-a107-c9b3c51b210d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e07052-2f47-420f-a107-c9b3c51b210d-plugins-config-volume\") pod \"30e07052-2f47-420f-a107-c9b3c51b210d\" (UID: \"30e07052-2f47-420f-a107-c9b3c51b210d\") " Oct 31 03:22:05.326939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.326913 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e07052-2f47-420f-a107-c9b3c51b210d-plugins-config-volume" pod "30e07052-2f47-420f-a107-c9b3c51b210d" (UID: "30e07052-2f47-420f-a107-c9b3c51b210d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:05.327118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.327091 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e07052-2f47-420f-a107-c9b3c51b210d-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:05.328615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.328596 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e07052-2f47-420f-a107-c9b3c51b210d-kube-api-access-mmkgk" pod "30e07052-2f47-420f-a107-c9b3c51b210d" (UID: "30e07052-2f47-420f-a107-c9b3c51b210d"). InnerVolumeSpecName "kube-api-access-mmkgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:05.428826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.428747 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmkgk\" (UniqueName: \"kubernetes.io/projected/30e07052-2f47-420f-a107-c9b3c51b210d-kube-api-access-mmkgk\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:05.443995 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.443964 4456 generic.go:356] "Generic (PLEG): container finished" podID="30e07052-2f47-420f-a107-c9b3c51b210d" containerID="bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce" exitCode=0 Oct 31 03:22:05.444160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.444015 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" Oct 31 03:22:05.444160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.444065 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" event={"ID":"30e07052-2f47-420f-a107-c9b3c51b210d","Type":"ContainerDied","Data":"bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce"} Oct 31 03:22:05.444160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.444106 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg" event={"ID":"30e07052-2f47-420f-a107-c9b3c51b210d","Type":"ContainerDied","Data":"a4a85c1a6fbd7c86056b85fc7fa5879e54766982b9ca25ed774b809908090094"} Oct 31 03:22:05.444160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.444127 4456 scope.go:122] "RemoveContainer" containerID="bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce" Oct 31 03:22:05.456650 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.456628 4456 scope.go:122] "RemoveContainer" containerID="bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce" Oct 31 03:22:05.456930 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:22:05.456904 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce\": container with ID starting with bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce not found: ID does not exist" containerID="bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce" Oct 31 03:22:05.456979 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.456948 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce"} err="failed to get container status \"bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce\": rpc error: code = NotFound desc = could not find container \"bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce\": container with ID starting with bb111268bc6d1f2e778a62d7333021c4f5e0d3ba68271abfea2f7a5bb6ecffce not found: ID does not exist" Oct 31 03:22:05.473234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.473207 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg"] Oct 31 03:22:05.477635 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:05.477611 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-677281-6848b58f85-bqzcg"] Oct 31 03:22:06.437083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:06.437030 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e07052-2f47-420f-a107-c9b3c51b210d" path="/var/lib/kubelet/pods/30e07052-2f47-420f-a107-c9b3c51b210d/volumes" Oct 31 03:22:07.038701 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:07.038669 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s"] Oct 31 03:22:07.043058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:07.043016 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863735a9-9mx5s"] Oct 31 03:22:08.439825 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:08.439789 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c62f1ec-97fb-4f46-991c-9088c99e11aa" path="/var/lib/kubelet/pods/5c62f1ec-97fb-4f46-991c-9088c99e11aa/volumes" Oct 31 03:22:09.219401 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.219363 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk"] Oct 31 03:22:09.220219 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220195 4456 cpu_manager.go:414] "containerMap: removing container" podUID="186c3940-33eb-42f6-93e2-def354123519" containerName="payload-pre-processing" Oct 31 03:22:09.220219 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220217 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186c3940-33eb-42f6-93e2-def354123519" containerName="payload-pre-processing" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220244 4456 cpu_manager.go:414] "containerMap: removing container" podUID="934b7bb6-89b4-4cd3-b746-037be2725481" containerName="revoke-keys" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220252 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934b7bb6-89b4-4cd3-b746-037be2725481" containerName="revoke-keys" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220273 4456 cpu_manager.go:414] "containerMap: removing container" podUID="2dcf78bf-69a6-42b8-99c9-ee5f51323973" containerName="payload-processing" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220281 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dcf78bf-69a6-42b8-99c9-ee5f51323973" containerName="payload-processing" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220294 4456 cpu_manager.go:414] "containerMap: removing container" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220302 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220319 4456 cpu_manager.go:414] "containerMap: removing container" podUID="37c79a8e-59ee-42e6-a4dc-f065a28a7a2d" containerName="revoke-keys" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220323 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c79a8e-59ee-42e6-a4dc-f065a28a7a2d" containerName="revoke-keys" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220342 4456 cpu_manager.go:414] "containerMap: removing container" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220347 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220360 4456 cpu_manager.go:414] "containerMap: removing container" podUID="30e07052-2f47-420f-a107-c9b3c51b210d" containerName="payload-processing" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220365 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e07052-2f47-420f-a107-c9b3c51b210d" containerName="payload-processing" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220380 4456 cpu_manager.go:414] "containerMap: removing container" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" Oct 31 03:22:09.220394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220384 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" Oct 31 03:22:09.220807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220485 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc17f71-5f77-48e0-a0c9-395bb51b4388" containerName="istio-proxy" Oct 31 03:22:09.220807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220493 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="186c3940-33eb-42f6-93e2-def354123519" containerName="payload-pre-processing" Oct 31 03:22:09.220807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220501 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="934b7bb6-89b4-4cd3-b746-037be2725481" containerName="revoke-keys" Oct 31 03:22:09.220807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220509 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dcf78bf-69a6-42b8-99c9-ee5f51323973" containerName="payload-processing" Oct 31 03:22:09.220807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220514 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d087ee5-7714-4cc3-88a8-de6e13297913" containerName="istio-proxy" Oct 31 03:22:09.220807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220523 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c79a8e-59ee-42e6-a4dc-f065a28a7a2d" containerName="revoke-keys" Oct 31 03:22:09.220807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220534 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="4657dd4e-e0de-4400-8304-74c7794421f5" containerName="istio-proxy" Oct 31 03:22:09.220807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.220543 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e07052-2f47-420f-a107-c9b3c51b210d" containerName="payload-processing" Oct 31 03:22:09.226298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.226273 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.246307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.246275 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk"] Oct 31 03:22:09.311017 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.310986 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-677281-openshift-default-dockercfg-2vqqr\"" Oct 31 03:22:09.392931 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.392876 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.393134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.392938 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.393134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.393104 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.393239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.393179 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.393239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.393220 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.393316 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.393254 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccmzz\" (UniqueName: \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-kube-api-access-ccmzz\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.393359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.393323 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.393359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.393350 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.393535 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.393435 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.494752 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.494660 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccmzz\" (UniqueName: \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-kube-api-access-ccmzz\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.494756 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.494786 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.494868 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.494913 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.494945 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.495115 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.495181 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.495218 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.495297 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.495346 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.495388 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.495577 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.495993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.495970 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.497454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.497426 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.497637 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.497620 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.507667 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.507644 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.507868 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.507851 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccmzz\" (UniqueName: \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-kube-api-access-ccmzz\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.548087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.548059 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:09.851214 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:22:09.851180 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47756ba9_12e5_4ada_b8c2_1a759f20f55e.slice/crio-d8324b92807a760309494162c25579e55ab5c4e1091638158f9bed81c32cc41d WatchSource:0}: Error finding container d8324b92807a760309494162c25579e55ab5c4e1091638158f9bed81c32cc41d: Status 404 returned error can't find the container with id d8324b92807a760309494162c25579e55ab5c4e1091638158f9bed81c32cc41d Oct 31 03:22:09.859653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.856246 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk"] Oct 31 03:22:09.859653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.858303 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:22:09.859653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.858469 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:22:09.859653 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:09.858562 4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 03:22:10.499134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:10.499097 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" event={"ID":"47756ba9-12e5-4ada-b8c2-1a759f20f55e","Type":"ContainerStarted","Data":"313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a"} Oct 31 03:22:10.499630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:10.499159 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" event={"ID":"47756ba9-12e5-4ada-b8c2-1a759f20f55e","Type":"ContainerStarted","Data":"d8324b92807a760309494162c25579e55ab5c4e1091638158f9bed81c32cc41d"} Oct 31 03:22:10.526590 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:10.526541 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" podStartSLOduration=1.526526897 podStartE2EDuration="1.526526897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:10.524921071 +0000 UTC m=+1612.205447199" watchObservedRunningTime="2026-10-31 03:22:10.526526897 +0000 UTC m=+1612.207053057" Oct 31 03:22:10.549225 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:10.549179 4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:10.551158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:10.551132 4456 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:22:10.551301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:10.551180 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" podUID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:22:11.557061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:11.550395 4456 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:22:11.557061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:11.550454 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" podUID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:22:12.558062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:12.553422 4456 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:22:12.558062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:12.553482 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" podUID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:22:13.561062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:13.550393 4456 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:22:13.561062 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:13.550464 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" podUID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:22:14.570056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:14.569353 4456 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:22:14.570056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:14.569409 4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" podUID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:22:14.629105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:14.626875 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk"] Oct 31 03:22:14.629105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:14.627169 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" podUID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" containerName="istio-proxy" containerID="cri-o://313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a" gracePeriod=30 Oct 31 03:22:15.487770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.487733 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn"] Oct 31 03:22:15.492386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.492356 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:15.510840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.510809 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn"] Oct 31 03:22:15.537025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.535542 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:22:15.537025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.535937 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:22:15.586003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.585490 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:15.586003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.585734 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdkfh\" (UniqueName: \"kubernetes.io/projected/c72daeca-25d3-4bea-92f2-b3f2263af2cb-kube-api-access-wdkfh\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:15.586003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.585811 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:15.690700 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.690652 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdkfh\" (UniqueName: \"kubernetes.io/projected/c72daeca-25d3-4bea-92f2-b3f2263af2cb-kube-api-access-wdkfh\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:15.690994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.690974 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:15.691168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.691150 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:15.699447 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.699417 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:15.699679 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.699473 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:15.713280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.713246 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdkfh\" (UniqueName: \"kubernetes.io/projected/c72daeca-25d3-4bea-92f2-b3f2263af2cb-kube-api-access-wdkfh\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:15.812934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:15.812852 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:16.147892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:16.147851 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn"] Oct 31 03:22:16.151128 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:22:16.151081 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72daeca_25d3_4bea_92f2_b3f2263af2cb.slice/crio-87b9103fe8ad4f7730180a5cc619c6ee207fa4718160e1d24e481a3f3dddf8a1 WatchSource:0}: Error finding container 87b9103fe8ad4f7730180a5cc619c6ee207fa4718160e1d24e481a3f3dddf8a1: Status 404 returned error can't find the container with id 87b9103fe8ad4f7730180a5cc619c6ee207fa4718160e1d24e481a3f3dddf8a1 Oct 31 03:22:16.608117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:16.608067 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" event={"ID":"c72daeca-25d3-4bea-92f2-b3f2263af2cb","Type":"ContainerStarted","Data":"87b9103fe8ad4f7730180a5cc619c6ee207fa4718160e1d24e481a3f3dddf8a1"} Oct 31 03:22:17.624560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:17.624454 4456 generic.go:356] "Generic (PLEG): container finished" podID="c72daeca-25d3-4bea-92f2-b3f2263af2cb" containerID="5525adefd7479e8a5bf04cd9fbbe7c8a38d08a7c20371cb871ba2c844f7c1a22" exitCode=0 Oct 31 03:22:17.624560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:17.624529 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" event={"ID":"c72daeca-25d3-4bea-92f2-b3f2263af2cb","Type":"ContainerDied","Data":"5525adefd7479e8a5bf04cd9fbbe7c8a38d08a7c20371cb871ba2c844f7c1a22"} Oct 31 03:22:18.797141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:18.797110 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:18.838555 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:18.838519 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72daeca-25d3-4bea-92f2-b3f2263af2cb-kube-api-access-wdkfh\" (UniqueName: \"kubernetes.io/projected/c72daeca-25d3-4bea-92f2-b3f2263af2cb-kube-api-access-wdkfh\") pod \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " Oct 31 03:22:18.838742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:18.838568 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-xks-ca\") pod \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " Oct 31 03:22:18.838742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:18.838704 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-openshift-ca\") pod \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\" (UID: \"c72daeca-25d3-4bea-92f2-b3f2263af2cb\") " Oct 31 03:22:18.839382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:18.839346 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-openshift-ca" pod "c72daeca-25d3-4bea-92f2-b3f2263af2cb" (UID: "c72daeca-25d3-4bea-92f2-b3f2263af2cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:18.840850 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:18.840827 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72daeca-25d3-4bea-92f2-b3f2263af2cb-kube-api-access-wdkfh" pod "c72daeca-25d3-4bea-92f2-b3f2263af2cb" (UID: "c72daeca-25d3-4bea-92f2-b3f2263af2cb"). InnerVolumeSpecName "kube-api-access-wdkfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:18.841137 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:18.841102 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-xks-ca" pod "c72daeca-25d3-4bea-92f2-b3f2263af2cb" (UID: "c72daeca-25d3-4bea-92f2-b3f2263af2cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:18.940479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:18.940440 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdkfh\" (UniqueName: \"kubernetes.io/projected/c72daeca-25d3-4bea-92f2-b3f2263af2cb-kube-api-access-wdkfh\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:18.940479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:18.940474 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:18.940479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:18.940483 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72daeca-25d3-4bea-92f2-b3f2263af2cb-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:19.344475 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:19.344393 4456 scope.go:122] "RemoveContainer" containerID="c1e5b833c405fae66a87598676258667032d924bc0a2c8cac82ddacd3bbf95e2" Oct 31 03:22:19.642534 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:19.642461 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" Oct 31 03:22:19.642534 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:19.642499 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn" event={"ID":"c72daeca-25d3-4bea-92f2-b3f2263af2cb","Type":"ContainerDied","Data":"87b9103fe8ad4f7730180a5cc619c6ee207fa4718160e1d24e481a3f3dddf8a1"} Oct 31 03:22:19.642736 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:19.642545 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87b9103fe8ad4f7730180a5cc619c6ee207fa4718160e1d24e481a3f3dddf8a1" Oct 31 03:22:20.134471 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:20.134435 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb"] Oct 31 03:22:20.151742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:20.151705 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv"] Oct 31 03:22:21.523142 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.523076 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.578683 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-kube-api-access-ccmzz\" (UniqueName: \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-kube-api-access-ccmzz\") pod \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.578748 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-token\" (UniqueName: \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-token\") pod \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.578863 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-certs\") pod \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.578929 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-credential-socket\") pod \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.578990 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istiod-ca-cert\") pod \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.579087 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-data\") pod \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.579183 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-socket\") pod \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.579295 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-envoy\") pod \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.579322 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-podinfo\") pod \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\" (UID: \"47756ba9-12e5-4ada-b8c2-1a759f20f55e\") " Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.580871 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istiod-ca-cert" pod "47756ba9-12e5-4ada-b8c2-1a759f20f55e" (UID: "47756ba9-12e5-4ada-b8c2-1a759f20f55e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.581156 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-data" pod "47756ba9-12e5-4ada-b8c2-1a759f20f55e" (UID: "47756ba9-12e5-4ada-b8c2-1a759f20f55e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.581385 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-socket" pod "47756ba9-12e5-4ada-b8c2-1a759f20f55e" (UID: "47756ba9-12e5-4ada-b8c2-1a759f20f55e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.581958 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-certs" pod "47756ba9-12e5-4ada-b8c2-1a759f20f55e" (UID: "47756ba9-12e5-4ada-b8c2-1a759f20f55e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.587107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.582248 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-credential-socket" pod "47756ba9-12e5-4ada-b8c2-1a759f20f55e" (UID: "47756ba9-12e5-4ada-b8c2-1a759f20f55e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.598582 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.598400 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-podinfo" pod "47756ba9-12e5-4ada-b8c2-1a759f20f55e" (UID: "47756ba9-12e5-4ada-b8c2-1a759f20f55e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:22:21.611500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.611472 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-envoy" pod "47756ba9-12e5-4ada-b8c2-1a759f20f55e" (UID: "47756ba9-12e5-4ada-b8c2-1a759f20f55e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.611630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.611522 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-kube-api-access-ccmzz" pod "47756ba9-12e5-4ada-b8c2-1a759f20f55e" (UID: "47756ba9-12e5-4ada-b8c2-1a759f20f55e"). InnerVolumeSpecName "kube-api-access-ccmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:21.611630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.611552 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-token" pod "47756ba9-12e5-4ada-b8c2-1a759f20f55e" (UID: "47756ba9-12e5-4ada-b8c2-1a759f20f55e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:21.671059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.670583 4456 generic.go:356] "Generic (PLEG): container finished" podID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" containerID="313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a" exitCode=0 Oct 31 03:22:21.671059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.670646 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" Oct 31 03:22:21.671059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.670695 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" event={"ID":"47756ba9-12e5-4ada-b8c2-1a759f20f55e","Type":"ContainerDied","Data":"313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a"} Oct 31 03:22:21.671059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.670745 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk" event={"ID":"47756ba9-12e5-4ada-b8c2-1a759f20f55e","Type":"ContainerDied","Data":"d8324b92807a760309494162c25579e55ab5c4e1091638158f9bed81c32cc41d"} Oct 31 03:22:21.671059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.670773 4456 scope.go:122] "RemoveContainer" containerID="313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a" Oct 31 03:22:21.682265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.682111 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.682265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.682141 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.682265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.682156 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.682265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.682169 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccmzz\" (UniqueName: \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-kube-api-access-ccmzz\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.682265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.682183 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.682265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.682196 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.682265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.682209 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.682265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.682222 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.682265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.682236 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47756ba9-12e5-4ada-b8c2-1a759f20f55e-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.702197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.702169 4456 scope.go:122] "RemoveContainer" containerID="313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a" Oct 31 03:22:21.702805 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:22:21.702774 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a\": container with ID starting with 313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a not found: ID does not exist" containerID="313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a" Oct 31 03:22:21.702936 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.702814 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a"} err="failed to get container status \"313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a\": rpc error: code = NotFound desc = could not find container \"313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a\": container with ID starting with 313a3ba58ea3908d81e850ea49f8d785301a3768c56b0754464bf008b8027b5a not found: ID does not exist" Oct 31 03:22:21.708129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.708099 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk"] Oct 31 03:22:21.727515 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:21.727491 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-xw4gk"] Oct 31 03:22:22.463065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:22.462363 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" path="/var/lib/kubelet/pods/47756ba9-12e5-4ada-b8c2-1a759f20f55e/volumes" Oct 31 03:22:22.566582 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:22.566545 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw"] Oct 31 03:22:22.570796 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:22.570745 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbbc81-cmzxw"] Oct 31 03:22:24.436678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:24.436641 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="934b7bb6-89b4-4cd3-b746-037be2725481" path="/var/lib/kubelet/pods/934b7bb6-89b4-4cd3-b746-037be2725481/volumes" Oct 31 03:22:32.137140 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:22:32.137103 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:22:32.847568 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.847528 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l"] Oct 31 03:22:32.848540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.848520 4456 cpu_manager.go:414] "containerMap: removing container" podUID="c72daeca-25d3-4bea-92f2-b3f2263af2cb" containerName="revoke-keys" Oct 31 03:22:32.848600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.848544 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72daeca-25d3-4bea-92f2-b3f2263af2cb" containerName="revoke-keys" Oct 31 03:22:32.848600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.848566 4456 cpu_manager.go:414] "containerMap: removing container" podUID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" containerName="istio-proxy" Oct 31 03:22:32.848600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.848573 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" containerName="istio-proxy" Oct 31 03:22:32.848757 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.848744 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="47756ba9-12e5-4ada-b8c2-1a759f20f55e" containerName="istio-proxy" Oct 31 03:22:32.848799 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.848777 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72daeca-25d3-4bea-92f2-b3f2263af2cb" containerName="revoke-keys" Oct 31 03:22:32.854770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.854749 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:32.859389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.859358 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:22:32.861363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.861344 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:22:32.868910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.868863 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l"] Oct 31 03:22:32.965865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.965830 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgw27\" (UniqueName: \"kubernetes.io/projected/5352217f-d494-4909-bb89-8ad1911118d3-kube-api-access-lgw27\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:32.965865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.965865 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:32.966144 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:32.966071 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:33.005349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.005313 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp"] Oct 31 03:22:33.009916 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.009894 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb"] Oct 31 03:22:33.010030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.010011 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.014463 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.014431 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.024594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.024546 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp"] Oct 31 03:22:33.030690 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.030652 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb"] Oct 31 03:22:33.066982 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.066939 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgw27\" (UniqueName: \"kubernetes.io/projected/5352217f-d494-4909-bb89-8ad1911118d3-kube-api-access-lgw27\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:33.067175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.067001 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:33.067175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.067140 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:33.067838 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.067810 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:33.069470 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.069448 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:33.080320 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.080298 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgw27\" (UniqueName: \"kubernetes.io/projected/5352217f-d494-4909-bb89-8ad1911118d3-kube-api-access-lgw27\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:33.168252 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.168217 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85cf461a-8pwjp\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.168698 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.168371 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56b17858-z4sjb\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.168698 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.168402 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69dsv\" (UniqueName: \"kubernetes.io/projected/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-kube-api-access-69dsv\") pod \"maas-api-revoke-sub-56b17858-z4sjb\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.168698 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.168590 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85cf461a-8pwjp\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.168890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.168698 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56b17858-z4sjb\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.168890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.168806 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkn46\" (UniqueName: \"kubernetes.io/projected/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-kube-api-access-tkn46\") pod \"maas-api-revoke-sub-85cf461a-8pwjp\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.171876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.171849 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:33.271570 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.270654 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85cf461a-8pwjp\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.271570 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.270724 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56b17858-z4sjb\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.271570 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.270770 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkn46\" (UniqueName: \"kubernetes.io/projected/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-kube-api-access-tkn46\") pod \"maas-api-revoke-sub-85cf461a-8pwjp\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.271570 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.270806 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85cf461a-8pwjp\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.271570 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.270875 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56b17858-z4sjb\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.271570 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.270907 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69dsv\" (UniqueName: \"kubernetes.io/projected/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-kube-api-access-69dsv\") pod \"maas-api-revoke-sub-56b17858-z4sjb\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.272048 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.271902 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85cf461a-8pwjp\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.272631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.272595 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56b17858-z4sjb\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.274396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.274370 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56b17858-z4sjb\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.275174 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.275138 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85cf461a-8pwjp\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.284346 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.284325 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69dsv\" (UniqueName: \"kubernetes.io/projected/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-kube-api-access-69dsv\") pod \"maas-api-revoke-sub-56b17858-z4sjb\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.284797 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.284775 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkn46\" (UniqueName: \"kubernetes.io/projected/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-kube-api-access-tkn46\") pod \"maas-api-revoke-sub-85cf461a-8pwjp\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.329652 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.329609 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:33.348819 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.348789 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:33.457738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.457699 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l"] Oct 31 03:22:33.650223 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:22:33.650196 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b7118d_9b9e_4ab0_856f_f69fe5149e5e.slice/crio-ee3310176b185352620435d90bb3eeee582b09cd1ec0011237cbb36968b860a0 WatchSource:0}: Error finding container ee3310176b185352620435d90bb3eeee582b09cd1ec0011237cbb36968b860a0: Status 404 returned error can't find the container with id ee3310176b185352620435d90bb3eeee582b09cd1ec0011237cbb36968b860a0 Oct 31 03:22:33.651216 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:22:33.651190 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe6bbf7_3e18_4ce8_97d7_a8510bcd44df.slice/crio-652c14b2127e88aaeb40aaf95a6ad53f5b71474adb96f190764027d4ff543d8a WatchSource:0}: Error finding container 652c14b2127e88aaeb40aaf95a6ad53f5b71474adb96f190764027d4ff543d8a: Status 404 returned error can't find the container with id 652c14b2127e88aaeb40aaf95a6ad53f5b71474adb96f190764027d4ff543d8a Oct 31 03:22:33.651883 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.651860 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp"] Oct 31 03:22:33.657389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.657359 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb"] Oct 31 03:22:33.804594 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.804262 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" event={"ID":"13b7118d-9b9e-4ab0-856f-f69fe5149e5e","Type":"ContainerStarted","Data":"ee3310176b185352620435d90bb3eeee582b09cd1ec0011237cbb36968b860a0"} Oct 31 03:22:33.806351 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.806323 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" event={"ID":"5352217f-d494-4909-bb89-8ad1911118d3","Type":"ContainerStarted","Data":"2c6d421e50937eb5402b13a646c8f8fe133a29671accfa932549ebb085904e2a"} Oct 31 03:22:33.808188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:33.808165 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" event={"ID":"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df","Type":"ContainerStarted","Data":"652c14b2127e88aaeb40aaf95a6ad53f5b71474adb96f190764027d4ff543d8a"} Oct 31 03:22:34.821395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:34.821362 4456 generic.go:356] "Generic (PLEG): container finished" podID="13b7118d-9b9e-4ab0-856f-f69fe5149e5e" containerID="912127e1072d7207aab13870b1b7b0ee766c9aab981b3d9cb77a6626cf5ea9a1" exitCode=0 Oct 31 03:22:34.821748 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:34.821449 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" event={"ID":"13b7118d-9b9e-4ab0-856f-f69fe5149e5e","Type":"ContainerDied","Data":"912127e1072d7207aab13870b1b7b0ee766c9aab981b3d9cb77a6626cf5ea9a1"} Oct 31 03:22:34.824069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:34.824004 4456 generic.go:356] "Generic (PLEG): container finished" podID="5352217f-d494-4909-bb89-8ad1911118d3" containerID="5046f0be351fc21e57246b03e70aef8a1b2c257f9c2945ae0cb2e576411b0e89" exitCode=0 Oct 31 03:22:34.824210 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:34.824074 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" event={"ID":"5352217f-d494-4909-bb89-8ad1911118d3","Type":"ContainerDied","Data":"5046f0be351fc21e57246b03e70aef8a1b2c257f9c2945ae0cb2e576411b0e89"} Oct 31 03:22:34.826364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:34.826342 4456 generic.go:356] "Generic (PLEG): container finished" podID="abe6bbf7-3e18-4ce8-97d7-a8510bcd44df" containerID="9acad260bb0b2f9e2a0430f0cbc603da54ff1537c6398728b73f52be2fcb7908" exitCode=0 Oct 31 03:22:34.826492 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:34.826386 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" event={"ID":"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df","Type":"ContainerDied","Data":"9acad260bb0b2f9e2a0430f0cbc603da54ff1537c6398728b73f52be2fcb7908"} Oct 31 03:22:35.157709 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.157673 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" containerID="cri-o://a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431" gracePeriod=45 Oct 31 03:22:35.182105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.182067 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" containerID="cri-o://e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff" gracePeriod=45 Oct 31 03:22:35.495501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.495462 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:22:35.519410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.519368 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:22:35.608066 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608007 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-home\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-home\") pod \"ab9ee55b-39a4-48da-a929-20cbb3f41730\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " Oct 31 03:22:35.608268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608082 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0661629-6e95-40c9-a8d1-4646ae2634f9-kube-api-access-swfrn\" (UniqueName: \"kubernetes.io/projected/a0661629-6e95-40c9-a8d1-4646ae2634f9-kube-api-access-swfrn\") pod \"a0661629-6e95-40c9-a8d1-4646ae2634f9\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " Oct 31 03:22:35.608268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608127 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-tmp-dir\") pod \"ab9ee55b-39a4-48da-a929-20cbb3f41730\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " Oct 31 03:22:35.608398 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608266 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-dshm\") pod \"ab9ee55b-39a4-48da-a929-20cbb3f41730\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " Oct 31 03:22:35.608398 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608332 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-home\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-home\") pod \"a0661629-6e95-40c9-a8d1-4646ae2634f9\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " Oct 31 03:22:35.608398 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608373 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-tmp-dir\") pod \"a0661629-6e95-40c9-a8d1-4646ae2634f9\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " Oct 31 03:22:35.608564 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608418 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-tmp-dir" pod "ab9ee55b-39a4-48da-a929-20cbb3f41730" (UID: "ab9ee55b-39a4-48da-a929-20cbb3f41730"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.608564 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608459 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-dshm\") pod \"a0661629-6e95-40c9-a8d1-4646ae2634f9\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " Oct 31 03:22:35.608678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608576 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0661629-6e95-40c9-a8d1-4646ae2634f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0661629-6e95-40c9-a8d1-4646ae2634f9-tls-certs\") pod \"a0661629-6e95-40c9-a8d1-4646ae2634f9\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " Oct 31 03:22:35.608678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608599 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab9ee55b-39a4-48da-a929-20cbb3f41730-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ee55b-39a4-48da-a929-20cbb3f41730-tls-certs\") pod \"ab9ee55b-39a4-48da-a929-20cbb3f41730\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " Oct 31 03:22:35.608678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608633 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-model-cache\") pod \"a0661629-6e95-40c9-a8d1-4646ae2634f9\" (UID: \"a0661629-6e95-40c9-a8d1-4646ae2634f9\") " Oct 31 03:22:35.608830 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608693 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-model-cache\") pod \"ab9ee55b-39a4-48da-a929-20cbb3f41730\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " Oct 31 03:22:35.608830 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608729 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-home" pod "a0661629-6e95-40c9-a8d1-4646ae2634f9" (UID: "a0661629-6e95-40c9-a8d1-4646ae2634f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.608830 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.608740 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9ee55b-39a4-48da-a929-20cbb3f41730-kube-api-access-5xtb9\" (UniqueName: \"kubernetes.io/projected/ab9ee55b-39a4-48da-a929-20cbb3f41730-kube-api-access-5xtb9\") pod \"ab9ee55b-39a4-48da-a929-20cbb3f41730\" (UID: \"ab9ee55b-39a4-48da-a929-20cbb3f41730\") " Oct 31 03:22:35.609387 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.609278 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-model-cache" pod "ab9ee55b-39a4-48da-a929-20cbb3f41730" (UID: "ab9ee55b-39a4-48da-a929-20cbb3f41730"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.609518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.609411 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-home" pod "ab9ee55b-39a4-48da-a929-20cbb3f41730" (UID: "ab9ee55b-39a4-48da-a929-20cbb3f41730"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.609518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.609478 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.609518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.609499 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.609518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.609513 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.609832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.609796 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-tmp-dir" pod "a0661629-6e95-40c9-a8d1-4646ae2634f9" (UID: "a0661629-6e95-40c9-a8d1-4646ae2634f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.609913 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.609864 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-model-cache" pod "a0661629-6e95-40c9-a8d1-4646ae2634f9" (UID: "a0661629-6e95-40c9-a8d1-4646ae2634f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.611191 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.611169 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-dshm" pod "ab9ee55b-39a4-48da-a929-20cbb3f41730" (UID: "ab9ee55b-39a4-48da-a929-20cbb3f41730"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.611284 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.611238 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0661629-6e95-40c9-a8d1-4646ae2634f9-kube-api-access-swfrn" pod "a0661629-6e95-40c9-a8d1-4646ae2634f9" (UID: "a0661629-6e95-40c9-a8d1-4646ae2634f9"). InnerVolumeSpecName "kube-api-access-swfrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:35.611679 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.611661 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9ee55b-39a4-48da-a929-20cbb3f41730-kube-api-access-5xtb9" pod "ab9ee55b-39a4-48da-a929-20cbb3f41730" (UID: "ab9ee55b-39a4-48da-a929-20cbb3f41730"). InnerVolumeSpecName "kube-api-access-5xtb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:35.611817 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.611795 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-dshm" pod "a0661629-6e95-40c9-a8d1-4646ae2634f9" (UID: "a0661629-6e95-40c9-a8d1-4646ae2634f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.612201 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.612185 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0661629-6e95-40c9-a8d1-4646ae2634f9-tls-certs" pod "a0661629-6e95-40c9-a8d1-4646ae2634f9" (UID: "a0661629-6e95-40c9-a8d1-4646ae2634f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:35.612685 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.612665 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab9ee55b-39a4-48da-a929-20cbb3f41730-tls-certs" pod "ab9ee55b-39a4-48da-a929-20cbb3f41730" (UID: "ab9ee55b-39a4-48da-a929-20cbb3f41730"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:35.711440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.711353 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.711440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.711393 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtb9\" (UniqueName: \"kubernetes.io/projected/ab9ee55b-39a4-48da-a929-20cbb3f41730-kube-api-access-5xtb9\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.711440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.711410 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.711440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.711423 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swfrn\" (UniqueName: \"kubernetes.io/projected/a0661629-6e95-40c9-a8d1-4646ae2634f9-kube-api-access-swfrn\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.711440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.711436 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9ee55b-39a4-48da-a929-20cbb3f41730-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.711712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.711451 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.711712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.711463 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0661629-6e95-40c9-a8d1-4646ae2634f9-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.711712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.711476 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0661629-6e95-40c9-a8d1-4646ae2634f9-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.711712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.711490 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ee55b-39a4-48da-a929-20cbb3f41730-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.864250 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.864210 4456 generic.go:356] "Generic (PLEG): container finished" podID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerID="a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431" exitCode=0 Oct 31 03:22:35.864747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.864362 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" Oct 31 03:22:35.864747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.864557 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" event={"ID":"ab9ee55b-39a4-48da-a929-20cbb3f41730","Type":"ContainerDied","Data":"a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431"} Oct 31 03:22:35.864747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.864600 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb" event={"ID":"ab9ee55b-39a4-48da-a929-20cbb3f41730","Type":"ContainerDied","Data":"01d1a4a7a9673f1a15df2ef4f1b9b98c0a69f981fa3a0ef5c660ed7d06b1999d"} Oct 31 03:22:35.864747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.864625 4456 scope.go:122] "RemoveContainer" containerID="a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431" Oct 31 03:22:35.867680 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.867650 4456 generic.go:356] "Generic (PLEG): container finished" podID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerID="e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff" exitCode=0 Oct 31 03:22:35.868075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.868022 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" Oct 31 03:22:35.868178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.868151 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" event={"ID":"a0661629-6e95-40c9-a8d1-4646ae2634f9","Type":"ContainerDied","Data":"e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff"} Oct 31 03:22:35.868239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.868198 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv" event={"ID":"a0661629-6e95-40c9-a8d1-4646ae2634f9","Type":"ContainerDied","Data":"b00ac30a17e71e5f5c4cb5e59ca17acdc9aaf8595949fce8dfbbdc44ef20d2d9"} Oct 31 03:22:35.894155 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.892839 4456 scope.go:122] "RemoveContainer" containerID="a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431" Oct 31 03:22:35.895338 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:22:35.895120 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431\": container with ID starting with a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431 not found: ID does not exist" containerID="a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431" Oct 31 03:22:35.895338 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.895164 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431"} err="failed to get container status \"a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431\": rpc error: code = NotFound desc = could not find container \"a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431\": container with ID starting with a6d16bdf473e2037920d6d794c99bb03d9f191a8a8649002ae0439354a823431 not found: ID does not exist" Oct 31 03:22:35.895338 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.895190 4456 scope.go:122] "RemoveContainer" containerID="e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff" Oct 31 03:22:35.915127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.913913 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb"] Oct 31 03:22:35.926280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.922812 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-w6-7d8daf-kserve-7bcd9754f9-njknb"] Oct 31 03:22:35.932160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.931762 4456 scope.go:122] "RemoveContainer" containerID="e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff" Oct 31 03:22:35.932558 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:22:35.932534 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff\": container with ID starting with e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff not found: ID does not exist" containerID="e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff" Oct 31 03:22:35.932718 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.932686 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff"} err="failed to get container status \"e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff\": rpc error: code = NotFound desc = could not find container \"e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff\": container with ID starting with e47513adf3c66b2104f7c20cc914801f6075fe260a043691a2936f5d1f6453ff not found: ID does not exist" Oct 31 03:22:35.935405 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.935374 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv"] Oct 31 03:22:35.942529 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:35.942502 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/simulator-w6-7d8daf-kserve-f68f57cdf-z2jtv"] Oct 31 03:22:36.064255 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.064211 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:36.127817 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.127788 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:36.128625 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.128600 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5352217f-d494-4909-bb89-8ad1911118d3-kube-api-access-lgw27\" (UniqueName: \"kubernetes.io/projected/5352217f-d494-4909-bb89-8ad1911118d3-kube-api-access-lgw27\") pod \"5352217f-d494-4909-bb89-8ad1911118d3\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " Oct 31 03:22:36.128856 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.128839 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-xks-ca\") pod \"5352217f-d494-4909-bb89-8ad1911118d3\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " Oct 31 03:22:36.128926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.128910 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-openshift-ca\") pod \"5352217f-d494-4909-bb89-8ad1911118d3\" (UID: \"5352217f-d494-4909-bb89-8ad1911118d3\") " Oct 31 03:22:36.129606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.129361 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-openshift-ca" pod "5352217f-d494-4909-bb89-8ad1911118d3" (UID: "5352217f-d494-4909-bb89-8ad1911118d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:36.129606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.129576 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:36.135203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.135175 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-xks-ca" pod "5352217f-d494-4909-bb89-8ad1911118d3" (UID: "5352217f-d494-4909-bb89-8ad1911118d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:36.135376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.135355 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5352217f-d494-4909-bb89-8ad1911118d3-kube-api-access-lgw27" pod "5352217f-d494-4909-bb89-8ad1911118d3" (UID: "5352217f-d494-4909-bb89-8ad1911118d3"). InnerVolumeSpecName "kube-api-access-lgw27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:36.213512 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.213485 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:36.230949 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.230919 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-xks-ca\") pod \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " Oct 31 03:22:36.231098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.230954 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-kube-api-access-tkn46\" (UniqueName: \"kubernetes.io/projected/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-kube-api-access-tkn46\") pod \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " Oct 31 03:22:36.231098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.231000 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-kube-api-access-69dsv\" (UniqueName: \"kubernetes.io/projected/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-kube-api-access-69dsv\") pod \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " Oct 31 03:22:36.231098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.231093 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-openshift-ca\") pod \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " Oct 31 03:22:36.231245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.231119 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-xks-ca\") pod \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\" (UID: \"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df\") " Oct 31 03:22:36.231245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.231156 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-openshift-ca\") pod \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\" (UID: \"13b7118d-9b9e-4ab0-856f-f69fe5149e5e\") " Oct 31 03:22:36.231519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.231480 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-openshift-ca" pod "abe6bbf7-3e18-4ce8-97d7-a8510bcd44df" (UID: "abe6bbf7-3e18-4ce8-97d7-a8510bcd44df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:36.231616 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.231589 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-openshift-ca" pod "13b7118d-9b9e-4ab0-856f-f69fe5149e5e" (UID: "13b7118d-9b9e-4ab0-856f-f69fe5149e5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:36.231873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.231848 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgw27\" (UniqueName: \"kubernetes.io/projected/5352217f-d494-4909-bb89-8ad1911118d3-kube-api-access-lgw27\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:36.232015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.231880 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:36.232015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.231922 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:36.232015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.231941 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5352217f-d494-4909-bb89-8ad1911118d3-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:36.233161 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.233142 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-xks-ca" pod "13b7118d-9b9e-4ab0-856f-f69fe5149e5e" (UID: "13b7118d-9b9e-4ab0-856f-f69fe5149e5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:36.233333 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.233311 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-xks-ca" pod "abe6bbf7-3e18-4ce8-97d7-a8510bcd44df" (UID: "abe6bbf7-3e18-4ce8-97d7-a8510bcd44df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:36.233640 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.233620 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-kube-api-access-69dsv" pod "abe6bbf7-3e18-4ce8-97d7-a8510bcd44df" (UID: "abe6bbf7-3e18-4ce8-97d7-a8510bcd44df"). InnerVolumeSpecName "kube-api-access-69dsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:36.233724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.233652 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-kube-api-access-tkn46" pod "13b7118d-9b9e-4ab0-856f-f69fe5149e5e" (UID: "13b7118d-9b9e-4ab0-856f-f69fe5149e5e"). InnerVolumeSpecName "kube-api-access-tkn46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:36.333784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.333697 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:36.333784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.333740 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkn46\" (UniqueName: \"kubernetes.io/projected/13b7118d-9b9e-4ab0-856f-f69fe5149e5e-kube-api-access-tkn46\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:36.333784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.333758 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69dsv\" (UniqueName: \"kubernetes.io/projected/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-kube-api-access-69dsv\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:36.333784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.333774 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:36.438345 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.438313 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" path="/var/lib/kubelet/pods/a0661629-6e95-40c9-a8d1-4646ae2634f9/volumes" Oct 31 03:22:36.438773 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.438760 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" path="/var/lib/kubelet/pods/ab9ee55b-39a4-48da-a929-20cbb3f41730/volumes" Oct 31 03:22:36.881839 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.881802 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" Oct 31 03:22:36.881839 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.881834 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb" event={"ID":"abe6bbf7-3e18-4ce8-97d7-a8510bcd44df","Type":"ContainerDied","Data":"652c14b2127e88aaeb40aaf95a6ad53f5b71474adb96f190764027d4ff543d8a"} Oct 31 03:22:36.882321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.881918 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="652c14b2127e88aaeb40aaf95a6ad53f5b71474adb96f190764027d4ff543d8a" Oct 31 03:22:36.885702 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.885668 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" event={"ID":"13b7118d-9b9e-4ab0-856f-f69fe5149e5e","Type":"ContainerDied","Data":"ee3310176b185352620435d90bb3eeee582b09cd1ec0011237cbb36968b860a0"} Oct 31 03:22:36.885904 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.885719 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee3310176b185352620435d90bb3eeee582b09cd1ec0011237cbb36968b860a0" Oct 31 03:22:36.886000 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.885980 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp" Oct 31 03:22:36.887379 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.887358 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" Oct 31 03:22:36.887610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.887437 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l" event={"ID":"5352217f-d494-4909-bb89-8ad1911118d3","Type":"ContainerDied","Data":"2c6d421e50937eb5402b13a646c8f8fe133a29671accfa932549ebb085904e2a"} Oct 31 03:22:36.887610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:36.887480 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c6d421e50937eb5402b13a646c8f8fe133a29671accfa932549ebb085904e2a" Oct 31 03:22:42.977084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:42.971933 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj"] Oct 31 03:22:42.997865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:42.997828 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79"] Oct 31 03:22:47.973364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:47.973298 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" podUID="a2f9db23-301e-4897-848e-6e1a1582b5d3" containerName="payload-processing" containerID="cri-o://77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74" gracePeriod=25 Oct 31 03:22:47.998952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:47.998896 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" podUID="b9f44798-81be-45f0-ad96-c56832446ef6" containerName="payload-pre-processing" containerID="cri-o://5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847" gracePeriod=25 Oct 31 03:22:48.291747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.291678 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:22:48.369175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.369141 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:22:48.424323 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.424278 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f9db23-301e-4897-848e-6e1a1582b5d3-kube-api-access-pzcgd\" (UniqueName: \"kubernetes.io/projected/a2f9db23-301e-4897-848e-6e1a1582b5d3-kube-api-access-pzcgd\") pod \"a2f9db23-301e-4897-848e-6e1a1582b5d3\" (UID: \"a2f9db23-301e-4897-848e-6e1a1582b5d3\") " Oct 31 03:22:48.424494 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.424357 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f9db23-301e-4897-848e-6e1a1582b5d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f9db23-301e-4897-848e-6e1a1582b5d3-plugins-config-volume\") pod \"a2f9db23-301e-4897-848e-6e1a1582b5d3\" (UID: \"a2f9db23-301e-4897-848e-6e1a1582b5d3\") " Oct 31 03:22:48.424833 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.424805 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f9db23-301e-4897-848e-6e1a1582b5d3-plugins-config-volume" pod "a2f9db23-301e-4897-848e-6e1a1582b5d3" (UID: "a2f9db23-301e-4897-848e-6e1a1582b5d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:48.425131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.425108 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f9db23-301e-4897-848e-6e1a1582b5d3-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:48.426642 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.426612 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f9db23-301e-4897-848e-6e1a1582b5d3-kube-api-access-pzcgd" pod "a2f9db23-301e-4897-848e-6e1a1582b5d3" (UID: "a2f9db23-301e-4897-848e-6e1a1582b5d3"). InnerVolumeSpecName "kube-api-access-pzcgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:48.525882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.525798 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f44798-81be-45f0-ad96-c56832446ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f44798-81be-45f0-ad96-c56832446ef6-plugins-config-volume\") pod \"b9f44798-81be-45f0-ad96-c56832446ef6\" (UID: \"b9f44798-81be-45f0-ad96-c56832446ef6\") " Oct 31 03:22:48.525882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.525860 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f44798-81be-45f0-ad96-c56832446ef6-kube-api-access-tbkl6\" (UniqueName: \"kubernetes.io/projected/b9f44798-81be-45f0-ad96-c56832446ef6-kube-api-access-tbkl6\") pod \"b9f44798-81be-45f0-ad96-c56832446ef6\" (UID: \"b9f44798-81be-45f0-ad96-c56832446ef6\") " Oct 31 03:22:48.526316 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.526290 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f44798-81be-45f0-ad96-c56832446ef6-plugins-config-volume" pod "b9f44798-81be-45f0-ad96-c56832446ef6" (UID: "b9f44798-81be-45f0-ad96-c56832446ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:48.526552 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.526535 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f44798-81be-45f0-ad96-c56832446ef6-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:48.526591 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.526555 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzcgd\" (UniqueName: \"kubernetes.io/projected/a2f9db23-301e-4897-848e-6e1a1582b5d3-kube-api-access-pzcgd\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:48.528094 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.528068 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f44798-81be-45f0-ad96-c56832446ef6-kube-api-access-tbkl6" pod "b9f44798-81be-45f0-ad96-c56832446ef6" (UID: "b9f44798-81be-45f0-ad96-c56832446ef6"). InnerVolumeSpecName "kube-api-access-tbkl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:48.627941 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:48.627901 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkl6\" (UniqueName: \"kubernetes.io/projected/b9f44798-81be-45f0-ad96-c56832446ef6-kube-api-access-tbkl6\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:49.004405 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.004372 4456 generic.go:356] "Generic (PLEG): container finished" podID="b9f44798-81be-45f0-ad96-c56832446ef6" containerID="5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847" exitCode=0 Oct 31 03:22:49.004810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.004413 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" Oct 31 03:22:49.004810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.004454 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" event={"ID":"b9f44798-81be-45f0-ad96-c56832446ef6","Type":"ContainerDied","Data":"5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847"} Oct 31 03:22:49.004810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.004510 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79" event={"ID":"b9f44798-81be-45f0-ad96-c56832446ef6","Type":"ContainerDied","Data":"f698765b073ea1f59f72d882196ce4d541dae342adb9090b529b67d50922ba51"} Oct 31 03:22:49.004810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.004541 4456 scope.go:122] "RemoveContainer" containerID="5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847" Oct 31 03:22:49.006302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.006269 4456 generic.go:356] "Generic (PLEG): container finished" podID="a2f9db23-301e-4897-848e-6e1a1582b5d3" containerID="77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74" exitCode=0 Oct 31 03:22:49.006421 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.006312 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" Oct 31 03:22:49.006421 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.006315 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" event={"ID":"a2f9db23-301e-4897-848e-6e1a1582b5d3","Type":"ContainerDied","Data":"77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74"} Oct 31 03:22:49.006421 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.006362 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj" event={"ID":"a2f9db23-301e-4897-848e-6e1a1582b5d3","Type":"ContainerDied","Data":"bf90f012a01091f8b6e236d43f478af240852948f937a9426c245b83a6c8d87c"} Oct 31 03:22:49.038101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.038072 4456 scope.go:122] "RemoveContainer" containerID="5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847" Oct 31 03:22:49.038639 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:22:49.038612 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847\": container with ID starting with 5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847 not found: ID does not exist" containerID="5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847" Oct 31 03:22:49.038762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.038647 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847"} err="failed to get container status \"5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847\": rpc error: code = NotFound desc = could not find container \"5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847\": container with ID starting with 5f1af0ef22a7a724a3d06ab00bf37790481892b0872c83ae1860dcf92c4c1847 not found: ID does not exist" Oct 31 03:22:49.038762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.038664 4456 scope.go:122] "RemoveContainer" containerID="77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74" Oct 31 03:22:49.043342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.043320 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj"] Oct 31 03:22:49.050218 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.050192 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7692c4-5c6d67d995-sqsnj"] Oct 31 03:22:49.053270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.053251 4456 scope.go:122] "RemoveContainer" containerID="77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74" Oct 31 03:22:49.053906 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:22:49.053880 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74\": container with ID starting with 77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74 not found: ID does not exist" containerID="77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74" Oct 31 03:22:49.054012 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.053916 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74"} err="failed to get container status \"77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74\": rpc error: code = NotFound desc = could not find container \"77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74\": container with ID starting with 77b94ce69ceed949f47c17431577b08260320275d1bec7960d77c57a09590b74 not found: ID does not exist" Oct 31 03:22:49.058003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.057971 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79"] Oct 31 03:22:49.063871 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:49.063847 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7692c4-75788d6fcf-zhx79"] Oct 31 03:22:50.454716 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:50.454677 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f9db23-301e-4897-848e-6e1a1582b5d3" path="/var/lib/kubelet/pods/a2f9db23-301e-4897-848e-6e1a1582b5d3/volumes" Oct 31 03:22:50.457897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:50.457869 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f44798-81be-45f0-ad96-c56832446ef6" path="/var/lib/kubelet/pods/b9f44798-81be-45f0-ad96-c56832446ef6/volumes" Oct 31 03:22:50.874112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:50.874016 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l"] Oct 31 03:22:50.880697 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:50.880666 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7692c4-gmr2l"] Oct 31 03:22:52.436830 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:52.436794 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5352217f-d494-4909-bb89-8ad1911118d3" path="/var/lib/kubelet/pods/5352217f-d494-4909-bb89-8ad1911118d3/volumes" Oct 31 03:22:56.031073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:56.031019 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f"] Oct 31 03:22:56.036083 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:56.036050 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cf5ce7-gm29f"] Oct 31 03:22:56.453060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:56.448236 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257a8229-2fdf-4467-9360-b0d6a822dfb1" path="/var/lib/kubelet/pods/257a8229-2fdf-4467-9360-b0d6a822dfb1/volumes" Oct 31 03:22:56.453060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:56.448719 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn"] Oct 31 03:22:56.476061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:56.474281 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d8daf-jswmn"] Oct 31 03:22:58.034314 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:58.034273 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59"] Oct 31 03:22:58.039729 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:58.039700 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602b57f0-xxr59"] Oct 31 03:22:58.438146 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:58.438108 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72daeca-25d3-4bea-92f2-b3f2263af2cb" path="/var/lib/kubelet/pods/c72daeca-25d3-4bea-92f2-b3f2263af2cb/volumes" Oct 31 03:22:58.438752 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:22:58.438729 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5287f4d-6337-4712-b9b9-bcc96d614cee" path="/var/lib/kubelet/pods/f5287f4d-6337-4712-b9b9-bcc96d614cee/volumes" Oct 31 03:23:01.825915 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.825831 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp"] Oct 31 03:23:01.826500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826482 4456 cpu_manager.go:414] "containerMap: removing container" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" Oct 31 03:23:01.826500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826499 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826512 4456 cpu_manager.go:414] "containerMap: removing container" podUID="13b7118d-9b9e-4ab0-856f-f69fe5149e5e" containerName="revoke-keys" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826517 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b7118d-9b9e-4ab0-856f-f69fe5149e5e" containerName="revoke-keys" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826528 4456 cpu_manager.go:414] "containerMap: removing container" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826535 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826546 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b9f44798-81be-45f0-ad96-c56832446ef6" containerName="payload-pre-processing" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826551 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f44798-81be-45f0-ad96-c56832446ef6" containerName="payload-pre-processing" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826558 4456 cpu_manager.go:414] "containerMap: removing container" podUID="a2f9db23-301e-4897-848e-6e1a1582b5d3" containerName="payload-processing" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826562 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f9db23-301e-4897-848e-6e1a1582b5d3" containerName="payload-processing" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826576 4456 cpu_manager.go:414] "containerMap: removing container" podUID="5352217f-d494-4909-bb89-8ad1911118d3" containerName="revoke-keys" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826581 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5352217f-d494-4909-bb89-8ad1911118d3" containerName="revoke-keys" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826597 4456 cpu_manager.go:414] "containerMap: removing container" podUID="abe6bbf7-3e18-4ce8-97d7-a8510bcd44df" containerName="revoke-keys" Oct 31 03:23:01.826615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826602 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe6bbf7-3e18-4ce8-97d7-a8510bcd44df" containerName="revoke-keys" Oct 31 03:23:01.827231 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826716 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe6bbf7-3e18-4ce8-97d7-a8510bcd44df" containerName="revoke-keys" Oct 31 03:23:01.827231 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826730 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0661629-6e95-40c9-a8d1-4646ae2634f9" containerName="main" Oct 31 03:23:01.827231 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826737 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b7118d-9b9e-4ab0-856f-f69fe5149e5e" containerName="revoke-keys" Oct 31 03:23:01.827231 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826745 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9ee55b-39a4-48da-a929-20cbb3f41730" containerName="main" Oct 31 03:23:01.827231 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826751 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f44798-81be-45f0-ad96-c56832446ef6" containerName="payload-pre-processing" Oct 31 03:23:01.827231 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826756 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f9db23-301e-4897-848e-6e1a1582b5d3" containerName="payload-processing" Oct 31 03:23:01.827231 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.826763 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5352217f-d494-4909-bb89-8ad1911118d3" containerName="revoke-keys" Oct 31 03:23:01.833696 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.832234 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:01.838774 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.838743 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp"] Oct 31 03:23:01.861082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.860400 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:23:01.861082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.860446 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:23:01.957287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.957235 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:01.957535 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.957507 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2db7\" (UniqueName: \"kubernetes.io/projected/7b9c6c76-f853-484f-8141-7478a4d3bcba-kube-api-access-p2db7\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:01.957625 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:01.957560 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:02.059369 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:02.059329 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2db7\" (UniqueName: \"kubernetes.io/projected/7b9c6c76-f853-484f-8141-7478a4d3bcba-kube-api-access-p2db7\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:02.059593 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:02.059391 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:02.059593 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:02.059469 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:02.060276 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:02.060246 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:02.062224 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:02.062197 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:02.072603 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:02.072571 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2db7\" (UniqueName: \"kubernetes.io/projected/7b9c6c76-f853-484f-8141-7478a4d3bcba-kube-api-access-p2db7\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:02.135017 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:23:02.134933 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:23:02.160370 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:02.160335 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:02.460304 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:23:02.460271 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9c6c76_f853_484f_8141_7478a4d3bcba.slice/crio-b3b13a1129b1a5728467210397c09c3834c2069eba8569a9370c175e5dd1d898 WatchSource:0}: Error finding container b3b13a1129b1a5728467210397c09c3834c2069eba8569a9370c175e5dd1d898: Status 404 returned error can't find the container with id b3b13a1129b1a5728467210397c09c3834c2069eba8569a9370c175e5dd1d898 Oct 31 03:23:02.460492 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:02.460470 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp"] Oct 31 03:23:03.211479 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:03.211443 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" event={"ID":"7b9c6c76-f853-484f-8141-7478a4d3bcba","Type":"ContainerStarted","Data":"1c9004cba6bdc1af0ea8c398e3ead4c6b6e7dee7383ea8b49ba9938b47869cd5"} Oct 31 03:23:03.211867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:03.211504 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" event={"ID":"7b9c6c76-f853-484f-8141-7478a4d3bcba","Type":"ContainerStarted","Data":"b3b13a1129b1a5728467210397c09c3834c2069eba8569a9370c175e5dd1d898"} Oct 31 03:23:03.234205 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:03.234157 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" podStartSLOduration=2.234141224 podStartE2EDuration="2.234141224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:03.231029063 +0000 UTC m=+1664.911555193" watchObservedRunningTime="2026-10-31 03:23:03.234141224 +0000 UTC m=+1664.914667351" Oct 31 03:23:04.221703 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:04.221665 4456 generic.go:356] "Generic (PLEG): container finished" podID="7b9c6c76-f853-484f-8141-7478a4d3bcba" containerID="1c9004cba6bdc1af0ea8c398e3ead4c6b6e7dee7383ea8b49ba9938b47869cd5" exitCode=0 Oct 31 03:23:04.222157 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:04.221748 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" event={"ID":"7b9c6c76-f853-484f-8141-7478a4d3bcba","Type":"ContainerDied","Data":"1c9004cba6bdc1af0ea8c398e3ead4c6b6e7dee7383ea8b49ba9938b47869cd5"} Oct 31 03:23:04.495134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:04.490145 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z"] Oct 31 03:23:04.495134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:04.490438 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" containerID="cri-o://9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c" gracePeriod=30 Oct 31 03:23:04.495715 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:04.495601 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5"] Oct 31 03:23:04.496026 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:04.495849 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" containerID="cri-o://312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc" gracePeriod=30 Oct 31 03:23:05.410864 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:05.410837 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:05.512408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:05.512322 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-xks-ca\") pod \"7b9c6c76-f853-484f-8141-7478a4d3bcba\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " Oct 31 03:23:05.512408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:05.512391 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9c6c76-f853-484f-8141-7478a4d3bcba-kube-api-access-p2db7\" (UniqueName: \"kubernetes.io/projected/7b9c6c76-f853-484f-8141-7478a4d3bcba-kube-api-access-p2db7\") pod \"7b9c6c76-f853-484f-8141-7478a4d3bcba\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " Oct 31 03:23:05.512604 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:05.512454 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-openshift-ca\") pod \"7b9c6c76-f853-484f-8141-7478a4d3bcba\" (UID: \"7b9c6c76-f853-484f-8141-7478a4d3bcba\") " Oct 31 03:23:05.512829 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:05.512800 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-openshift-ca" pod "7b9c6c76-f853-484f-8141-7478a4d3bcba" (UID: "7b9c6c76-f853-484f-8141-7478a4d3bcba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:05.513184 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:05.513154 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:05.514668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:05.514645 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-xks-ca" pod "7b9c6c76-f853-484f-8141-7478a4d3bcba" (UID: "7b9c6c76-f853-484f-8141-7478a4d3bcba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:05.514787 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:05.514669 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9c6c76-f853-484f-8141-7478a4d3bcba-kube-api-access-p2db7" pod "7b9c6c76-f853-484f-8141-7478a4d3bcba" (UID: "7b9c6c76-f853-484f-8141-7478a4d3bcba"). InnerVolumeSpecName "kube-api-access-p2db7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:05.614013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:05.613977 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9c6c76-f853-484f-8141-7478a4d3bcba-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:05.614013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:05.614007 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2db7\" (UniqueName: \"kubernetes.io/projected/7b9c6c76-f853-484f-8141-7478a4d3bcba-kube-api-access-p2db7\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:06.245502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:06.245465 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" event={"ID":"7b9c6c76-f853-484f-8141-7478a4d3bcba","Type":"ContainerDied","Data":"b3b13a1129b1a5728467210397c09c3834c2069eba8569a9370c175e5dd1d898"} Oct 31 03:23:06.245502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:06.245492 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp" Oct 31 03:23:06.245906 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:06.245514 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b13a1129b1a5728467210397c09c3834c2069eba8569a9370c175e5dd1d898" Oct 31 03:23:06.287251 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:23:06.287218 4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9c6c76_f853_484f_8141_7478a4d3bcba.slice\": RecentStats: unable to find data in memory cache]" Oct 31 03:23:07.027248 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:07.027206 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:23:07.027775 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:07.027269 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:23:07.069324 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:07.069280 4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:23:07.069486 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:07.069362 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:23:09.838727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.838699 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:23:09.870372 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.870063 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:23:09.983409 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983366 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-envoy\") pod \"fe931dea-f20e-4671-9eed-2d800466d2f0\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " Oct 31 03:23:09.983607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983437 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-certs\") pod \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " Oct 31 03:23:09.983607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983477 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-certs\") pod \"fe931dea-f20e-4671-9eed-2d800466d2f0\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " Oct 31 03:23:09.983607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983512 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-podinfo\") pod \"fe931dea-f20e-4671-9eed-2d800466d2f0\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " Oct 31 03:23:09.983607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983549 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-data\") pod \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " Oct 31 03:23:09.983607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983587 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-data\") pod \"fe931dea-f20e-4671-9eed-2d800466d2f0\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " Oct 31 03:23:09.983890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983653 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-kube-api-access-hdh7b\" (UniqueName: \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-kube-api-access-hdh7b\") pod \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " Oct 31 03:23:09.983890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983733 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-credential-socket\") pod \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " Oct 31 03:23:09.983890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983777 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-token\" (UniqueName: \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-token\") pod \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " Oct 31 03:23:09.983890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983828 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-token\" (UniqueName: \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-token\") pod \"fe931dea-f20e-4671-9eed-2d800466d2f0\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " Oct 31 03:23:09.983890 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983881 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-credential-socket\") pod \"fe931dea-f20e-4671-9eed-2d800466d2f0\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " Oct 31 03:23:09.984211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.983971 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-socket\") pod \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " Oct 31 03:23:09.984211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.984013 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-envoy\") pod \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " Oct 31 03:23:09.984211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.984090 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istiod-ca-cert\") pod \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " Oct 31 03:23:09.984211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.984175 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-socket\") pod \"fe931dea-f20e-4671-9eed-2d800466d2f0\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " Oct 31 03:23:09.984422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.984209 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-podinfo\") pod \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\" (UID: \"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5\") " Oct 31 03:23:09.984422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.984236 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe931dea-f20e-4671-9eed-2d800466d2f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe931dea-f20e-4671-9eed-2d800466d2f0-istiod-ca-cert\") pod \"fe931dea-f20e-4671-9eed-2d800466d2f0\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " Oct 31 03:23:09.984422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.984274 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-kube-api-access-2s954\" (UniqueName: \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-kube-api-access-2s954\") pod \"fe931dea-f20e-4671-9eed-2d800466d2f0\" (UID: \"fe931dea-f20e-4671-9eed-2d800466d2f0\") " Oct 31 03:23:09.984707 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.984662 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-certs" pod "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" (UID: "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.984780 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.984755 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-certs" pod "fe931dea-f20e-4671-9eed-2d800466d2f0" (UID: "fe931dea-f20e-4671-9eed-2d800466d2f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.985134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.985055 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985134 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.985086 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985375 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.985264 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-credential-socket" pod "fe931dea-f20e-4671-9eed-2d800466d2f0" (UID: "fe931dea-f20e-4671-9eed-2d800466d2f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.985476 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.985456 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-credential-socket" pod "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" (UID: "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.986611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.986079 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-socket" pod "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" (UID: "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.986771 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.986329 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-socket" pod "fe931dea-f20e-4671-9eed-2d800466d2f0" (UID: "fe931dea-f20e-4671-9eed-2d800466d2f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.986905 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.986475 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istiod-ca-cert" pod "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" (UID: "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:09.987075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.986634 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe931dea-f20e-4671-9eed-2d800466d2f0-istiod-ca-cert" pod "fe931dea-f20e-4671-9eed-2d800466d2f0" (UID: "fe931dea-f20e-4671-9eed-2d800466d2f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:09.987256 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.986649 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-data" pod "fe931dea-f20e-4671-9eed-2d800466d2f0" (UID: "fe931dea-f20e-4671-9eed-2d800466d2f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.987555 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.987529 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-kube-api-access-2s954" pod "fe931dea-f20e-4671-9eed-2d800466d2f0" (UID: "fe931dea-f20e-4671-9eed-2d800466d2f0"). InnerVolumeSpecName "kube-api-access-2s954". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:09.988654 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.988628 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-podinfo" pod "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" (UID: "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:23:09.989245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.989210 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-envoy" pod "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" (UID: "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.990376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.990351 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-token" pod "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" (UID: "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:09.990478 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.990456 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-kube-api-access-hdh7b" pod "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" (UID: "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5"). InnerVolumeSpecName "kube-api-access-hdh7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:09.990722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.990704 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-token" pod "fe931dea-f20e-4671-9eed-2d800466d2f0" (UID: "fe931dea-f20e-4671-9eed-2d800466d2f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:09.990961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.990931 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-podinfo" pod "fe931dea-f20e-4671-9eed-2d800466d2f0" (UID: "fe931dea-f20e-4671-9eed-2d800466d2f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:23:09.991097 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.991021 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-envoy" pod "fe931dea-f20e-4671-9eed-2d800466d2f0" (UID: "fe931dea-f20e-4671-9eed-2d800466d2f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.991172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:09.991149 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-data" pod "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" (UID: "da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:10.086682 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086593 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086682 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086632 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdh7b\" (UniqueName: \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-kube-api-access-hdh7b\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086682 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086649 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086682 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086665 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086682 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086679 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086692 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086707 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086720 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086733 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086746 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086759 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086773 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe931dea-f20e-4671-9eed-2d800466d2f0-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086790 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s954\" (UniqueName: \"kubernetes.io/projected/fe931dea-f20e-4671-9eed-2d800466d2f0-kube-api-access-2s954\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086803 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086816 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe931dea-f20e-4671-9eed-2d800466d2f0-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.086961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.086828 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.325561 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.325526 4456 generic.go:356] "Generic (PLEG): container finished" podID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerID="9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c" exitCode=0 Oct 31 03:23:10.325720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.325576 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" Oct 31 03:23:10.325720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.325608 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" event={"ID":"fe931dea-f20e-4671-9eed-2d800466d2f0","Type":"ContainerDied","Data":"9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c"} Oct 31 03:23:10.325720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.325685 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z" event={"ID":"fe931dea-f20e-4671-9eed-2d800466d2f0","Type":"ContainerDied","Data":"a91d58321a50d2828d142be04f911e5afc26c1fe3460104df02e14bb235ba142"} Oct 31 03:23:10.325720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.325711 4456 scope.go:122] "RemoveContainer" containerID="9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c" Oct 31 03:23:10.327512 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.327484 4456 generic.go:356] "Generic (PLEG): container finished" podID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerID="312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc" exitCode=0 Oct 31 03:23:10.327628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.327508 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" event={"ID":"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5","Type":"ContainerDied","Data":"312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc"} Oct 31 03:23:10.327628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.327555 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" event={"ID":"da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5","Type":"ContainerDied","Data":"3b21c758f78f06e7d306a43f6f5da2806a65f322f33cabdf17fbd900b08845b7"} Oct 31 03:23:10.327628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.327524 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5" Oct 31 03:23:10.338118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.338097 4456 scope.go:122] "RemoveContainer" containerID="9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c" Oct 31 03:23:10.338395 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:23:10.338371 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c\": container with ID starting with 9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c not found: ID does not exist" containerID="9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c" Oct 31 03:23:10.338464 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.338400 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c"} err="failed to get container status \"9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c\": rpc error: code = NotFound desc = could not find container \"9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c\": container with ID starting with 9c175d43d8609e31b82aa10d044ac157b589f7b0bdffd1ca6796b40fa3fa582c not found: ID does not exist" Oct 31 03:23:10.338464 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.338418 4456 scope.go:122] "RemoveContainer" containerID="312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc" Oct 31 03:23:10.349670 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.349649 4456 scope.go:122] "RemoveContainer" containerID="312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc" Oct 31 03:23:10.349950 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:23:10.349931 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc\": container with ID starting with 312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc not found: ID does not exist" containerID="312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc" Oct 31 03:23:10.350023 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.349958 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc"} err="failed to get container status \"312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc\": rpc error: code = NotFound desc = could not find container \"312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc\": container with ID starting with 312f4a022fd787748f6e2f91775a280bce3098f35f777f13a81d49542271d4fc not found: ID does not exist" Oct 31 03:23:10.357633 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.357612 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5"] Oct 31 03:23:10.365674 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.365650 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-mw5p5"] Oct 31 03:23:10.410675 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.405056 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z"] Oct 31 03:23:10.413085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.413057 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-p8p9z"] Oct 31 03:23:10.436619 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.436583 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" path="/var/lib/kubelet/pods/da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5/volumes" Oct 31 03:23:10.437211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:10.437190 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" path="/var/lib/kubelet/pods/fe931dea-f20e-4671-9eed-2d800466d2f0/volumes" Oct 31 03:23:12.056431 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:12.056391 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q"] Oct 31 03:23:15.026477 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:15.025457 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp"] Oct 31 03:23:15.041758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:15.041673 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3fc2d0-z6npp"] Oct 31 03:23:16.437427 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:16.437390 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9c6c76-f853-484f-8141-7478a4d3bcba" path="/var/lib/kubelet/pods/7b9c6c76-f853-484f-8141-7478a4d3bcba/volumes" Oct 31 03:23:17.058632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.058582 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" podUID="b0ad8b22-0439-463d-9d1e-518f50f7f7e4" containerName="payload-pre-processing" containerID="cri-o://be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0" gracePeriod=25 Oct 31 03:23:17.338357 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.338323 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:23:17.401029 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.400993 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-plugins-config-volume\") pod \"b0ad8b22-0439-463d-9d1e-518f50f7f7e4\" (UID: \"b0ad8b22-0439-463d-9d1e-518f50f7f7e4\") " Oct 31 03:23:17.401260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.401237 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-kube-api-access-pdcj4\" (UniqueName: \"kubernetes.io/projected/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-kube-api-access-pdcj4\") pod \"b0ad8b22-0439-463d-9d1e-518f50f7f7e4\" (UID: \"b0ad8b22-0439-463d-9d1e-518f50f7f7e4\") " Oct 31 03:23:17.401454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.401429 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-plugins-config-volume" pod "b0ad8b22-0439-463d-9d1e-518f50f7f7e4" (UID: "b0ad8b22-0439-463d-9d1e-518f50f7f7e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:17.401832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.401807 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:17.403457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.403434 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-kube-api-access-pdcj4" pod "b0ad8b22-0439-463d-9d1e-518f50f7f7e4" (UID: "b0ad8b22-0439-463d-9d1e-518f50f7f7e4"). InnerVolumeSpecName "kube-api-access-pdcj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:17.421664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.421634 4456 generic.go:356] "Generic (PLEG): container finished" podID="b0ad8b22-0439-463d-9d1e-518f50f7f7e4" containerID="be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0" exitCode=0 Oct 31 03:23:17.421758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.421679 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" Oct 31 03:23:17.421758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.421715 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" event={"ID":"b0ad8b22-0439-463d-9d1e-518f50f7f7e4","Type":"ContainerDied","Data":"be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0"} Oct 31 03:23:17.421840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.421773 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q" event={"ID":"b0ad8b22-0439-463d-9d1e-518f50f7f7e4","Type":"ContainerDied","Data":"d57a483fca5a50fee307168d0fc55cbdad638e7f8373c1f3c47490e81dc924ee"} Oct 31 03:23:17.421840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.421802 4456 scope.go:122] "RemoveContainer" containerID="be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0" Oct 31 03:23:17.439562 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.439542 4456 scope.go:122] "RemoveContainer" containerID="be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0" Oct 31 03:23:17.439876 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:23:17.439856 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0\": container with ID starting with be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0 not found: ID does not exist" containerID="be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0" Oct 31 03:23:17.439929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.439882 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0"} err="failed to get container status \"be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0\": rpc error: code = NotFound desc = could not find container \"be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0\": container with ID starting with be6e80813790069fbb189e4a96164b02cc7fcbb6923866155de68de5a2b453a0 not found: ID does not exist" Oct 31 03:23:17.450429 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.450406 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q"] Oct 31 03:23:17.454523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.454497 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3fc2d0-7774747b95-gcg9q"] Oct 31 03:23:17.502784 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:17.502751 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdcj4\" (UniqueName: \"kubernetes.io/projected/b0ad8b22-0439-463d-9d1e-518f50f7f7e4-kube-api-access-pdcj4\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:18.445373 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:18.445340 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ad8b22-0439-463d-9d1e-518f50f7f7e4" path="/var/lib/kubelet/pods/b0ad8b22-0439-463d-9d1e-518f50f7f7e4/volumes" Oct 31 03:23:19.453551 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:19.453514 4456 scope.go:122] "RemoveContainer" containerID="26baca26b51a5cdb8261be6c6ec18297f155b82012e3e223d7ff7c880ef7e746" Oct 31 03:23:19.467150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:19.467129 4456 scope.go:122] "RemoveContainer" containerID="01e57bf5f286b1a37071eba54480fa800c717ac33da138a852e0ab0cb6ccb0f5" Oct 31 03:23:19.477504 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:19.477479 4456 scope.go:122] "RemoveContainer" containerID="67b00222b91788090798f50a5561e588df911e1bb25e47bdd45d8dcec276d5c5" Oct 31 03:23:19.487616 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:19.487595 4456 scope.go:122] "RemoveContainer" containerID="fe56a063b78daa69a69c61dadfff36680fc7cb44845ffd2237bb72bf031950ab" Oct 31 03:23:24.486947 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.486912 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k"] Oct 31 03:23:24.487730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.487708 4456 cpu_manager.go:414] "containerMap: removing container" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" Oct 31 03:23:24.487837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.487731 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" Oct 31 03:23:24.487837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.487748 4456 cpu_manager.go:414] "containerMap: removing container" podUID="7b9c6c76-f853-484f-8141-7478a4d3bcba" containerName="revoke-keys" Oct 31 03:23:24.487837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.487755 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9c6c76-f853-484f-8141-7478a4d3bcba" containerName="revoke-keys" Oct 31 03:23:24.487837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.487799 4456 cpu_manager.go:414] "containerMap: removing container" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" Oct 31 03:23:24.487837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.487806 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" Oct 31 03:23:24.487837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.487819 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b0ad8b22-0439-463d-9d1e-518f50f7f7e4" containerName="payload-pre-processing" Oct 31 03:23:24.487837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.487826 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ad8b22-0439-463d-9d1e-518f50f7f7e4" containerName="payload-pre-processing" Oct 31 03:23:24.488232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.487983 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="da90e2ea-c695-4ce3-af2a-ad6f79d8b2e5" containerName="istio-proxy" Oct 31 03:23:24.488232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.487997 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ad8b22-0439-463d-9d1e-518f50f7f7e4" containerName="payload-pre-processing" Oct 31 03:23:24.488232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.488017 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe931dea-f20e-4671-9eed-2d800466d2f0" containerName="istio-proxy" Oct 31 03:23:24.488232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.488029 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9c6c76-f853-484f-8141-7478a4d3bcba" containerName="revoke-keys" Oct 31 03:23:24.493962 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.493938 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:24.500131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.500108 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:23:24.500315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.500162 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:23:24.502985 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.502957 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k"] Oct 31 03:23:24.632840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.632801 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:24.633015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.632938 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clczf\" (UniqueName: \"kubernetes.io/projected/8084c8fb-e70e-4476-b436-fd7173ac0e74-kube-api-access-clczf\") pod \"maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:24.633015 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.632994 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:24.734547 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.734512 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:24.734736 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.734599 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clczf\" (UniqueName: \"kubernetes.io/projected/8084c8fb-e70e-4476-b436-fd7173ac0e74-kube-api-access-clczf\") pod \"maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:24.734736 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.734641 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:24.735408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.735376 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:24.737350 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.737293 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:24.748293 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.748264 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clczf\" (UniqueName: \"kubernetes.io/projected/8084c8fb-e70e-4476-b436-fd7173ac0e74-kube-api-access-clczf\") pod \"maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:24.810879 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:24.810842 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:25.106946 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:23:25.106913 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8084c8fb_e70e_4476_b436_fd7173ac0e74.slice/crio-5ede403fa1ff366403bcd9fcb7cff01b2fc6e96ae57ad7a975820f40ec26a248 WatchSource:0}: Error finding container 5ede403fa1ff366403bcd9fcb7cff01b2fc6e96ae57ad7a975820f40ec26a248: Status 404 returned error can't find the container with id 5ede403fa1ff366403bcd9fcb7cff01b2fc6e96ae57ad7a975820f40ec26a248 Oct 31 03:23:25.107952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:25.107917 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k"] Oct 31 03:23:25.529086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:25.529015 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" event={"ID":"8084c8fb-e70e-4476-b436-fd7173ac0e74","Type":"ContainerStarted","Data":"3f8c5d09b44605606d997ca367b3a33d8145137942cdf33fd76b3dbbb112caf7"} Oct 31 03:23:25.529538 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:25.529115 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" event={"ID":"8084c8fb-e70e-4476-b436-fd7173ac0e74","Type":"ContainerStarted","Data":"5ede403fa1ff366403bcd9fcb7cff01b2fc6e96ae57ad7a975820f40ec26a248"} Oct 31 03:23:25.555274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:25.555226 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" podStartSLOduration=1.55521073 podStartE2EDuration="1.55521073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:25.551584628 +0000 UTC m=+1687.232110755" watchObservedRunningTime="2026-10-31 03:23:25.55521073 +0000 UTC m=+1687.235736851" Oct 31 03:23:26.540663 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:26.540627 4456 generic.go:356] "Generic (PLEG): container finished" podID="8084c8fb-e70e-4476-b436-fd7173ac0e74" containerID="3f8c5d09b44605606d997ca367b3a33d8145137942cdf33fd76b3dbbb112caf7" exitCode=0 Oct 31 03:23:26.541095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:26.540685 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" event={"ID":"8084c8fb-e70e-4476-b436-fd7173ac0e74","Type":"ContainerDied","Data":"3f8c5d09b44605606d997ca367b3a33d8145137942cdf33fd76b3dbbb112caf7"} Oct 31 03:23:27.734756 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:27.734727 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:27.881118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:27.881015 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-xks-ca\") pod \"8084c8fb-e70e-4476-b436-fd7173ac0e74\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " Oct 31 03:23:27.881272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:27.881204 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-openshift-ca\") pod \"8084c8fb-e70e-4476-b436-fd7173ac0e74\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " Oct 31 03:23:27.881322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:27.881292 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8084c8fb-e70e-4476-b436-fd7173ac0e74-kube-api-access-clczf\" (UniqueName: \"kubernetes.io/projected/8084c8fb-e70e-4476-b436-fd7173ac0e74-kube-api-access-clczf\") pod \"8084c8fb-e70e-4476-b436-fd7173ac0e74\" (UID: \"8084c8fb-e70e-4476-b436-fd7173ac0e74\") " Oct 31 03:23:27.881596 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:27.881563 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-openshift-ca" pod "8084c8fb-e70e-4476-b436-fd7173ac0e74" (UID: "8084c8fb-e70e-4476-b436-fd7173ac0e74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:27.881831 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:27.881805 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:27.883347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:27.883325 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8084c8fb-e70e-4476-b436-fd7173ac0e74-kube-api-access-clczf" pod "8084c8fb-e70e-4476-b436-fd7173ac0e74" (UID: "8084c8fb-e70e-4476-b436-fd7173ac0e74"). InnerVolumeSpecName "kube-api-access-clczf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:27.883347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:27.883333 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-xks-ca" pod "8084c8fb-e70e-4476-b436-fd7173ac0e74" (UID: "8084c8fb-e70e-4476-b436-fd7173ac0e74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:27.983577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:27.983540 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clczf\" (UniqueName: \"kubernetes.io/projected/8084c8fb-e70e-4476-b436-fd7173ac0e74-kube-api-access-clczf\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:27.983577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:27.983575 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8084c8fb-e70e-4476-b436-fd7173ac0e74-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:28.564137 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:28.564093 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" event={"ID":"8084c8fb-e70e-4476-b436-fd7173ac0e74","Type":"ContainerDied","Data":"5ede403fa1ff366403bcd9fcb7cff01b2fc6e96ae57ad7a975820f40ec26a248"} Oct 31 03:23:28.564312 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:28.564162 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ede403fa1ff366403bcd9fcb7cff01b2fc6e96ae57ad7a975820f40ec26a248" Oct 31 03:23:28.564312 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:28.564114 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k" Oct 31 03:23:29.251191 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:29.251155 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8"] Oct 31 03:23:31.711373 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.711332 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb"] Oct 31 03:23:31.712008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.711980 4456 cpu_manager.go:414] "containerMap: removing container" podUID="8084c8fb-e70e-4476-b436-fd7173ac0e74" containerName="revoke-keys" Oct 31 03:23:31.712008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.712008 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8084c8fb-e70e-4476-b436-fd7173ac0e74" containerName="revoke-keys" Oct 31 03:23:31.712189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.712158 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8084c8fb-e70e-4476-b436-fd7173ac0e74" containerName="revoke-keys" Oct 31 03:23:31.716013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.715991 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:31.723220 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.723194 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:23:31.723348 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.723306 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:23:31.724258 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.724235 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb"] Oct 31 03:23:31.730051 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.730005 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89e726e7-xp9sb\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:31.730162 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.730075 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgsxv\" (UniqueName: \"kubernetes.io/projected/34a88f55-d220-468a-8f3e-e1e3c36c628b-kube-api-access-dgsxv\") pod \"maas-api-revoke-sub-89e726e7-xp9sb\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:31.730283 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.730264 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89e726e7-xp9sb\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:31.831223 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.831191 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgsxv\" (UniqueName: \"kubernetes.io/projected/34a88f55-d220-468a-8f3e-e1e3c36c628b-kube-api-access-dgsxv\") pod \"maas-api-revoke-sub-89e726e7-xp9sb\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:31.831405 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.831306 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89e726e7-xp9sb\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:31.831508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.831489 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89e726e7-xp9sb\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:31.831978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.831959 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89e726e7-xp9sb\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:31.833744 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.833720 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89e726e7-xp9sb\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:31.844732 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:31.844710 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgsxv\" (UniqueName: \"kubernetes.io/projected/34a88f55-d220-468a-8f3e-e1e3c36c628b-kube-api-access-dgsxv\") pod \"maas-api-revoke-sub-89e726e7-xp9sb\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:32.034108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:32.033982 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:32.148160 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:23:32.148124 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:23:32.320450 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:23:32.320410 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a88f55_d220_468a_8f3e_e1e3c36c628b.slice/crio-6e0dd3722da60588fd22c1d4e6d92de185787c2b742220b0bd2ba9fdf2c6b6e5 WatchSource:0}: Error finding container 6e0dd3722da60588fd22c1d4e6d92de185787c2b742220b0bd2ba9fdf2c6b6e5: Status 404 returned error can't find the container with id 6e0dd3722da60588fd22c1d4e6d92de185787c2b742220b0bd2ba9fdf2c6b6e5 Oct 31 03:23:32.320690 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:32.320659 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb"] Oct 31 03:23:32.606275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:32.606240 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" event={"ID":"34a88f55-d220-468a-8f3e-e1e3c36c628b","Type":"ContainerStarted","Data":"6e0dd3722da60588fd22c1d4e6d92de185787c2b742220b0bd2ba9fdf2c6b6e5"} Oct 31 03:23:33.616696 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:33.616606 4456 generic.go:356] "Generic (PLEG): container finished" podID="34a88f55-d220-468a-8f3e-e1e3c36c628b" containerID="cdf3c2a0d032e612846a9eb68969a0a952be0c21ff8f0b021c04676a1c4a7b7f" exitCode=0 Oct 31 03:23:33.617102 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:33.616691 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" event={"ID":"34a88f55-d220-468a-8f3e-e1e3c36c628b","Type":"ContainerDied","Data":"cdf3c2a0d032e612846a9eb68969a0a952be0c21ff8f0b021c04676a1c4a7b7f"} Oct 31 03:23:34.855981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:34.855923 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:34.906220 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:34.906178 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-xks-ca\") pod \"34a88f55-d220-468a-8f3e-e1e3c36c628b\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " Oct 31 03:23:34.906408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:34.906358 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-openshift-ca\") pod \"34a88f55-d220-468a-8f3e-e1e3c36c628b\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " Oct 31 03:23:34.906748 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:34.906723 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-openshift-ca" pod "34a88f55-d220-468a-8f3e-e1e3c36c628b" (UID: "34a88f55-d220-468a-8f3e-e1e3c36c628b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:34.906879 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:34.906864 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a88f55-d220-468a-8f3e-e1e3c36c628b-kube-api-access-dgsxv\" (UniqueName: \"kubernetes.io/projected/34a88f55-d220-468a-8f3e-e1e3c36c628b-kube-api-access-dgsxv\") pod \"34a88f55-d220-468a-8f3e-e1e3c36c628b\" (UID: \"34a88f55-d220-468a-8f3e-e1e3c36c628b\") " Oct 31 03:23:34.914065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:34.907409 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:34.914065 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:34.909886 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a88f55-d220-468a-8f3e-e1e3c36c628b-kube-api-access-dgsxv" pod "34a88f55-d220-468a-8f3e-e1e3c36c628b" (UID: "34a88f55-d220-468a-8f3e-e1e3c36c628b"). InnerVolumeSpecName "kube-api-access-dgsxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:34.917719 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:34.917691 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-xks-ca" pod "34a88f55-d220-468a-8f3e-e1e3c36c628b" (UID: "34a88f55-d220-468a-8f3e-e1e3c36c628b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:35.009347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:35.009302 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a88f55-d220-468a-8f3e-e1e3c36c628b-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:35.009347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:35.009345 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgsxv\" (UniqueName: \"kubernetes.io/projected/34a88f55-d220-468a-8f3e-e1e3c36c628b-kube-api-access-dgsxv\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:35.637221 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:35.637189 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" Oct 31 03:23:35.637221 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:35.637212 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb" event={"ID":"34a88f55-d220-468a-8f3e-e1e3c36c628b","Type":"ContainerDied","Data":"6e0dd3722da60588fd22c1d4e6d92de185787c2b742220b0bd2ba9fdf2c6b6e5"} Oct 31 03:23:35.637468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:35.637268 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e0dd3722da60588fd22c1d4e6d92de185787c2b742220b0bd2ba9fdf2c6b6e5" Oct 31 03:23:40.954886 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:40.954841 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf"] Oct 31 03:23:40.955873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:40.955839 4456 cpu_manager.go:414] "containerMap: removing container" podUID="34a88f55-d220-468a-8f3e-e1e3c36c628b" containerName="revoke-keys" Oct 31 03:23:40.955873 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:40.955866 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a88f55-d220-468a-8f3e-e1e3c36c628b" containerName="revoke-keys" Oct 31 03:23:40.956148 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:40.956113 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a88f55-d220-468a-8f3e-e1e3c36c628b" containerName="revoke-keys" Oct 31 03:23:40.961637 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:40.961612 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:40.963203 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:40.963178 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj"] Oct 31 03:23:40.968383 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:40.968359 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:40.968528 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:40.968505 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:23:40.969977 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:40.969951 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf"] Oct 31 03:23:40.971362 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:40.971344 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:23:41.002875 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.002848 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj"] Oct 31 03:23:41.094146 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.094107 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09924894-xk2rj\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:41.094328 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.094302 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6cxb\" (UniqueName: \"kubernetes.io/projected/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-kube-api-access-j6cxb\") pod \"maas-api-revoke-sub-09924894-xk2rj\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:41.094391 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.094373 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c4c3764-kkdgf\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:41.094527 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.094500 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09924894-xk2rj\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:41.094613 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.094595 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c4c3764-kkdgf\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:41.094657 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.094642 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd44f\" (UniqueName: \"kubernetes.io/projected/f90285d8-ff96-45d7-949e-5fb48e0ed04a-kube-api-access-fd44f\") pod \"maas-api-revoke-sub-3c4c3764-kkdgf\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:41.195894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.195852 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6cxb\" (UniqueName: \"kubernetes.io/projected/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-kube-api-access-j6cxb\") pod \"maas-api-revoke-sub-09924894-xk2rj\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:41.196129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.196026 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c4c3764-kkdgf\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:41.196228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.196149 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09924894-xk2rj\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:41.196228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.196217 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c4c3764-kkdgf\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:41.196335 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.196265 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd44f\" (UniqueName: \"kubernetes.io/projected/f90285d8-ff96-45d7-949e-5fb48e0ed04a-kube-api-access-fd44f\") pod \"maas-api-revoke-sub-3c4c3764-kkdgf\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:41.196402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.196331 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09924894-xk2rj\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:41.196882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.196846 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c4c3764-kkdgf\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:41.197020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.196920 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09924894-xk2rj\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:41.198897 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.198869 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09924894-xk2rj\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:41.199009 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.198876 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c4c3764-kkdgf\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:41.209121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.209053 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6cxb\" (UniqueName: \"kubernetes.io/projected/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-kube-api-access-j6cxb\") pod \"maas-api-revoke-sub-09924894-xk2rj\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:41.209121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.209091 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd44f\" (UniqueName: \"kubernetes.io/projected/f90285d8-ff96-45d7-949e-5fb48e0ed04a-kube-api-access-fd44f\") pod \"maas-api-revoke-sub-3c4c3764-kkdgf\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:41.307474 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.303777 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:41.307474 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.306635 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:41.617411 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:23:41.617384 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af13e2f_3c49_4d1b_baa6_7a9b1959505c.slice/crio-ae5436dd8663a2080457edfda03a705316351227633818137f1af20f85be6f79 WatchSource:0}: Error finding container ae5436dd8663a2080457edfda03a705316351227633818137f1af20f85be6f79: Status 404 returned error can't find the container with id ae5436dd8663a2080457edfda03a705316351227633818137f1af20f85be6f79 Oct 31 03:23:41.618085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.618061 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj"] Oct 31 03:23:41.626245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.626223 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf"] Oct 31 03:23:41.629870 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:23:41.629836 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90285d8_ff96_45d7_949e_5fb48e0ed04a.slice/crio-20a82962f7bc12478cfc9df89e92e0a005c931336be6fb8d80fff1b44834bc90 WatchSource:0}: Error finding container 20a82962f7bc12478cfc9df89e92e0a005c931336be6fb8d80fff1b44834bc90: Status 404 returned error can't find the container with id 20a82962f7bc12478cfc9df89e92e0a005c931336be6fb8d80fff1b44834bc90 Oct 31 03:23:41.692573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.692537 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" event={"ID":"f90285d8-ff96-45d7-949e-5fb48e0ed04a","Type":"ContainerStarted","Data":"20a82962f7bc12478cfc9df89e92e0a005c931336be6fb8d80fff1b44834bc90"} Oct 31 03:23:41.694031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:41.693998 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" event={"ID":"9af13e2f-3c49-4d1b-baa6-7a9b1959505c","Type":"ContainerStarted","Data":"ae5436dd8663a2080457edfda03a705316351227633818137f1af20f85be6f79"} Oct 31 03:23:42.383118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:42.382749 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v"] Oct 31 03:23:42.710663 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:42.710624 4456 generic.go:356] "Generic (PLEG): container finished" podID="f90285d8-ff96-45d7-949e-5fb48e0ed04a" containerID="c9bc278c4ab312139c3049218eb97876209f07492b4c022cbd4a1740979e4d8e" exitCode=0 Oct 31 03:23:42.710824 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:42.710714 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" event={"ID":"f90285d8-ff96-45d7-949e-5fb48e0ed04a","Type":"ContainerDied","Data":"c9bc278c4ab312139c3049218eb97876209f07492b4c022cbd4a1740979e4d8e"} Oct 31 03:23:42.712525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:42.712500 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" event={"ID":"9af13e2f-3c49-4d1b-baa6-7a9b1959505c","Type":"ContainerStarted","Data":"7a1725912b303e5e9dfa140d5f1b7b3420f5a1ee42bac0a74c62dae6d71e97b7"} Oct 31 03:23:42.794656 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:42.794605 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" podStartSLOduration=2.794590168 podStartE2EDuration="2.794590168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:42.792353584 +0000 UTC m=+1704.472879714" watchObservedRunningTime="2026-10-31 03:23:42.794590168 +0000 UTC m=+1704.475116295" Oct 31 03:23:43.062115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.062020 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78"] Oct 31 03:23:43.106068 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.106009 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.110092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.110060 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78"] Oct 31 03:23:43.231912 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.231870 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2p7\" (UniqueName: \"kubernetes.io/projected/da158174-c1f6-4b39-a49a-ab75310ab817-kube-api-access-cm2p7\") pod \"maas-api-revoke-sub-3cf5c9d0-7rq78\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.232126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.231996 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf5c9d0-7rq78\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.232227 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.232201 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf5c9d0-7rq78\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.333473 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.333387 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf5c9d0-7rq78\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.333648 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.333523 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf5c9d0-7rq78\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.333721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.333645 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2p7\" (UniqueName: \"kubernetes.io/projected/da158174-c1f6-4b39-a49a-ab75310ab817-kube-api-access-cm2p7\") pod \"maas-api-revoke-sub-3cf5c9d0-7rq78\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.334305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.334277 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf5c9d0-7rq78\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.335851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.335831 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf5c9d0-7rq78\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.348510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.348483 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2p7\" (UniqueName: \"kubernetes.io/projected/da158174-c1f6-4b39-a49a-ab75310ab817-kube-api-access-cm2p7\") pod \"maas-api-revoke-sub-3cf5c9d0-7rq78\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.425222 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.425185 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:43.717726 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:23:43.717694 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda158174_c1f6_4b39_a49a_ab75310ab817.slice/crio-6064086e636f877f09385ea527e86b97a1ebca226f1c8961cb9b93c7879ca930 WatchSource:0}: Error finding container 6064086e636f877f09385ea527e86b97a1ebca226f1c8961cb9b93c7879ca930: Status 404 returned error can't find the container with id 6064086e636f877f09385ea527e86b97a1ebca226f1c8961cb9b93c7879ca930 Oct 31 03:23:43.719026 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.718992 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78"] Oct 31 03:23:43.727386 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.727355 4456 generic.go:356] "Generic (PLEG): container finished" podID="9af13e2f-3c49-4d1b-baa6-7a9b1959505c" containerID="7a1725912b303e5e9dfa140d5f1b7b3420f5a1ee42bac0a74c62dae6d71e97b7" exitCode=0 Oct 31 03:23:43.728167 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.728142 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" event={"ID":"9af13e2f-3c49-4d1b-baa6-7a9b1959505c","Type":"ContainerDied","Data":"7a1725912b303e5e9dfa140d5f1b7b3420f5a1ee42bac0a74c62dae6d71e97b7"} Oct 31 03:23:43.934920 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.934894 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:43.942045 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.942013 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-xks-ca\") pod \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " Oct 31 03:23:43.942218 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.942203 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90285d8-ff96-45d7-949e-5fb48e0ed04a-kube-api-access-fd44f\" (UniqueName: \"kubernetes.io/projected/f90285d8-ff96-45d7-949e-5fb48e0ed04a-kube-api-access-fd44f\") pod \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " Oct 31 03:23:43.942414 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.942395 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-openshift-ca\") pod \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\" (UID: \"f90285d8-ff96-45d7-949e-5fb48e0ed04a\") " Oct 31 03:23:43.942799 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.942771 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-openshift-ca" pod "f90285d8-ff96-45d7-949e-5fb48e0ed04a" (UID: "f90285d8-ff96-45d7-949e-5fb48e0ed04a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:43.943019 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.942996 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:43.944183 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.944163 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-xks-ca" pod "f90285d8-ff96-45d7-949e-5fb48e0ed04a" (UID: "f90285d8-ff96-45d7-949e-5fb48e0ed04a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:43.944269 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:43.944254 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90285d8-ff96-45d7-949e-5fb48e0ed04a-kube-api-access-fd44f" pod "f90285d8-ff96-45d7-949e-5fb48e0ed04a" (UID: "f90285d8-ff96-45d7-949e-5fb48e0ed04a"). InnerVolumeSpecName "kube-api-access-fd44f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:44.043811 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.043670 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90285d8-ff96-45d7-949e-5fb48e0ed04a-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.043811 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.043700 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd44f\" (UniqueName: \"kubernetes.io/projected/f90285d8-ff96-45d7-949e-5fb48e0ed04a-kube-api-access-fd44f\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.273320 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.273279 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" containerID="cri-o://c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47" gracePeriod=45 Oct 31 03:23:44.580001 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.579973 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:23:44.652632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.652595 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-home\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-home\") pod \"142a5e7e-e192-401a-9a93-a97d77e11f27\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " Oct 31 03:23:44.652832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.652653 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-dshm\") pod \"142a5e7e-e192-401a-9a93-a97d77e11f27\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " Oct 31 03:23:44.652832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.652713 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-tmp-dir\") pod \"142a5e7e-e192-401a-9a93-a97d77e11f27\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " Oct 31 03:23:44.652955 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.652879 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-model-cache\") pod \"142a5e7e-e192-401a-9a93-a97d77e11f27\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " Oct 31 03:23:44.652955 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.652899 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-home" pod "142a5e7e-e192-401a-9a93-a97d77e11f27" (UID: "142a5e7e-e192-401a-9a93-a97d77e11f27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.652955 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.652941 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/142a5e7e-e192-401a-9a93-a97d77e11f27-tls-certs\" (UniqueName: \"kubernetes.io/secret/142a5e7e-e192-401a-9a93-a97d77e11f27-tls-certs\") pod \"142a5e7e-e192-401a-9a93-a97d77e11f27\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " Oct 31 03:23:44.653219 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.653073 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142a5e7e-e192-401a-9a93-a97d77e11f27-kube-api-access-tl8cf\" (UniqueName: \"kubernetes.io/projected/142a5e7e-e192-401a-9a93-a97d77e11f27-kube-api-access-tl8cf\") pod \"142a5e7e-e192-401a-9a93-a97d77e11f27\" (UID: \"142a5e7e-e192-401a-9a93-a97d77e11f27\") " Oct 31 03:23:44.653219 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.653185 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-tmp-dir" pod "142a5e7e-e192-401a-9a93-a97d77e11f27" (UID: "142a5e7e-e192-401a-9a93-a97d77e11f27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.653742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.653649 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.653742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.653671 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.654168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.654113 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-model-cache" pod "142a5e7e-e192-401a-9a93-a97d77e11f27" (UID: "142a5e7e-e192-401a-9a93-a97d77e11f27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.659786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.656450 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142a5e7e-e192-401a-9a93-a97d77e11f27-tls-certs" pod "142a5e7e-e192-401a-9a93-a97d77e11f27" (UID: "142a5e7e-e192-401a-9a93-a97d77e11f27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:44.659786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.656561 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142a5e7e-e192-401a-9a93-a97d77e11f27-kube-api-access-tl8cf" pod "142a5e7e-e192-401a-9a93-a97d77e11f27" (UID: "142a5e7e-e192-401a-9a93-a97d77e11f27"). InnerVolumeSpecName "kube-api-access-tl8cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:44.665313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.665288 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-dshm" pod "142a5e7e-e192-401a-9a93-a97d77e11f27" (UID: "142a5e7e-e192-401a-9a93-a97d77e11f27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.741953 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.741907 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" Oct 31 03:23:44.742162 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.742017 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf" event={"ID":"f90285d8-ff96-45d7-949e-5fb48e0ed04a","Type":"ContainerDied","Data":"20a82962f7bc12478cfc9df89e92e0a005c931336be6fb8d80fff1b44834bc90"} Oct 31 03:23:44.742162 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.742083 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20a82962f7bc12478cfc9df89e92e0a005c931336be6fb8d80fff1b44834bc90" Oct 31 03:23:44.745540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.744245 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" event={"ID":"da158174-c1f6-4b39-a49a-ab75310ab817","Type":"ContainerStarted","Data":"12b4ff7ff4018f46f7cb6601ce180f082483359c89f4e1768a3a4ce6ddda5cc2"} Oct 31 03:23:44.745540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.744317 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" event={"ID":"da158174-c1f6-4b39-a49a-ab75310ab817","Type":"ContainerStarted","Data":"6064086e636f877f09385ea527e86b97a1ebca226f1c8961cb9b93c7879ca930"} Oct 31 03:23:44.748655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.747309 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" event={"ID":"142a5e7e-e192-401a-9a93-a97d77e11f27","Type":"ContainerDied","Data":"c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47"} Oct 31 03:23:44.748655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.747393 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" Oct 31 03:23:44.748655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.747514 4456 scope.go:122] "RemoveContainer" containerID="c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47" Oct 31 03:23:44.748910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.748690 4456 generic.go:356] "Generic (PLEG): container finished" podID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerID="c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47" exitCode=0 Oct 31 03:23:44.749441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.749415 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" event={"ID":"142a5e7e-e192-401a-9a93-a97d77e11f27","Type":"ContainerDied","Data":"b5aaa1dbd0b298500048977fbaf7c79edc740b86243e387ac4ece3cdd412ae8f"} Oct 31 03:23:44.764723 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.764643 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.764723 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.764677 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142a5e7e-e192-401a-9a93-a97d77e11f27-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.764723 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.764719 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl8cf\" (UniqueName: \"kubernetes.io/projected/142a5e7e-e192-401a-9a93-a97d77e11f27-kube-api-access-tl8cf\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.764944 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.764735 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142a5e7e-e192-401a-9a93-a97d77e11f27-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.774131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.774076 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" podStartSLOduration=1.77405789 podStartE2EDuration="1.77405789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:44.768975631 +0000 UTC m=+1706.449501757" watchObservedRunningTime="2026-10-31 03:23:44.77405789 +0000 UTC m=+1706.454584014" Oct 31 03:23:44.816708 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.816679 4456 scope.go:122] "RemoveContainer" containerID="c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47" Oct 31 03:23:44.818231 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:23:44.818203 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47\": container with ID starting with c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47 not found: ID does not exist" containerID="c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47" Oct 31 03:23:44.818412 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.818387 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47"} err="failed to get container status \"c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47\": rpc error: code = NotFound desc = could not find container \"c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47\": container with ID starting with c67b56c9eef267cd076d07000f2e56f496e067d38260d714c6a6b2f5be858c47 not found: ID does not exist" Oct 31 03:23:44.828131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.828104 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8"] Oct 31 03:23:44.834846 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.834816 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8"] Oct 31 03:23:44.923182 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.923151 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:44.967683 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.967648 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-kube-api-access-j6cxb\" (UniqueName: \"kubernetes.io/projected/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-kube-api-access-j6cxb\") pod \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " Oct 31 03:23:44.968451 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.967943 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-xks-ca\") pod \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " Oct 31 03:23:44.968451 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.968139 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-openshift-ca\") pod \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\" (UID: \"9af13e2f-3c49-4d1b-baa6-7a9b1959505c\") " Oct 31 03:23:44.968564 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.968482 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-openshift-ca" pod "9af13e2f-3c49-4d1b-baa6-7a9b1959505c" (UID: "9af13e2f-3c49-4d1b-baa6-7a9b1959505c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:44.968719 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.968697 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.970898 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.970872 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-kube-api-access-j6cxb" pod "9af13e2f-3c49-4d1b-baa6-7a9b1959505c" (UID: "9af13e2f-3c49-4d1b-baa6-7a9b1959505c"). InnerVolumeSpecName "kube-api-access-j6cxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:44.971008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:44.970902 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-xks-ca" pod "9af13e2f-3c49-4d1b-baa6-7a9b1959505c" (UID: "9af13e2f-3c49-4d1b-baa6-7a9b1959505c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:45.070084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:45.070027 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6cxb\" (UniqueName: \"kubernetes.io/projected/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-kube-api-access-j6cxb\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:45.070084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:45.070085 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af13e2f-3c49-4d1b-baa6-7a9b1959505c-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:45.530253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:45.530213 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/premium-w2-85cb59-kserve-5bbcc7775b-bz4x8" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:23:45.763249 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:45.763211 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" Oct 31 03:23:45.763249 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:45.763235 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj" event={"ID":"9af13e2f-3c49-4d1b-baa6-7a9b1959505c","Type":"ContainerDied","Data":"ae5436dd8663a2080457edfda03a705316351227633818137f1af20f85be6f79"} Oct 31 03:23:45.763804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:45.763296 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae5436dd8663a2080457edfda03a705316351227633818137f1af20f85be6f79" Oct 31 03:23:45.765063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:45.765018 4456 generic.go:356] "Generic (PLEG): container finished" podID="da158174-c1f6-4b39-a49a-ab75310ab817" containerID="12b4ff7ff4018f46f7cb6601ce180f082483359c89f4e1768a3a4ce6ddda5cc2" exitCode=0 Oct 31 03:23:45.765186 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:45.765069 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" event={"ID":"da158174-c1f6-4b39-a49a-ab75310ab817","Type":"ContainerDied","Data":"12b4ff7ff4018f46f7cb6601ce180f082483359c89f4e1768a3a4ce6ddda5cc2"} Oct 31 03:23:46.438031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:46.437998 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" path="/var/lib/kubelet/pods/142a5e7e-e192-401a-9a93-a97d77e11f27/volumes" Oct 31 03:23:46.963160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:46.963133 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:46.997615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:46.997193 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-openshift-ca\") pod \"da158174-c1f6-4b39-a49a-ab75310ab817\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " Oct 31 03:23:46.997801 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:46.997723 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-xks-ca\") pod \"da158174-c1f6-4b39-a49a-ab75310ab817\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " Oct 31 03:23:46.997926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:46.997866 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da158174-c1f6-4b39-a49a-ab75310ab817-kube-api-access-cm2p7\" (UniqueName: \"kubernetes.io/projected/da158174-c1f6-4b39-a49a-ab75310ab817-kube-api-access-cm2p7\") pod \"da158174-c1f6-4b39-a49a-ab75310ab817\" (UID: \"da158174-c1f6-4b39-a49a-ab75310ab817\") " Oct 31 03:23:46.999158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:46.999137 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-openshift-ca" pod "da158174-c1f6-4b39-a49a-ab75310ab817" (UID: "da158174-c1f6-4b39-a49a-ab75310ab817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:47.001531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.001503 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-xks-ca" pod "da158174-c1f6-4b39-a49a-ab75310ab817" (UID: "da158174-c1f6-4b39-a49a-ab75310ab817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:47.001531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.001514 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da158174-c1f6-4b39-a49a-ab75310ab817-kube-api-access-cm2p7" pod "da158174-c1f6-4b39-a49a-ab75310ab817" (UID: "da158174-c1f6-4b39-a49a-ab75310ab817"). InnerVolumeSpecName "kube-api-access-cm2p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:47.100082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.099984 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:47.100082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.100023 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm2p7\" (UniqueName: \"kubernetes.io/projected/da158174-c1f6-4b39-a49a-ab75310ab817-kube-api-access-cm2p7\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:47.100082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.100056 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da158174-c1f6-4b39-a49a-ab75310ab817-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:47.734138 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.734091 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4"] Oct 31 03:23:47.734762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.734742 4456 cpu_manager.go:414] "containerMap: removing container" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" Oct 31 03:23:47.734867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.734764 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" Oct 31 03:23:47.734867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.734813 4456 cpu_manager.go:414] "containerMap: removing container" podUID="f90285d8-ff96-45d7-949e-5fb48e0ed04a" containerName="revoke-keys" Oct 31 03:23:47.734867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.734822 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90285d8-ff96-45d7-949e-5fb48e0ed04a" containerName="revoke-keys" Oct 31 03:23:47.734867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.734843 4456 cpu_manager.go:414] "containerMap: removing container" podUID="da158174-c1f6-4b39-a49a-ab75310ab817" containerName="revoke-keys" Oct 31 03:23:47.734867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.734850 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da158174-c1f6-4b39-a49a-ab75310ab817" containerName="revoke-keys" Oct 31 03:23:47.735198 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.734868 4456 cpu_manager.go:414] "containerMap: removing container" podUID="9af13e2f-3c49-4d1b-baa6-7a9b1959505c" containerName="revoke-keys" Oct 31 03:23:47.735198 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.734876 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af13e2f-3c49-4d1b-baa6-7a9b1959505c" containerName="revoke-keys" Oct 31 03:23:47.735198 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.735028 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="da158174-c1f6-4b39-a49a-ab75310ab817" containerName="revoke-keys" Oct 31 03:23:47.735198 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.735062 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af13e2f-3c49-4d1b-baa6-7a9b1959505c" containerName="revoke-keys" Oct 31 03:23:47.735198 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.735078 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="142a5e7e-e192-401a-9a93-a97d77e11f27" containerName="main" Oct 31 03:23:47.735198 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.735095 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90285d8-ff96-45d7-949e-5fb48e0ed04a" containerName="revoke-keys" Oct 31 03:23:47.738891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.738870 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:47.747429 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.747406 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4"] Oct 31 03:23:47.784983 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.784952 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" Oct 31 03:23:47.784983 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.784961 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78" event={"ID":"da158174-c1f6-4b39-a49a-ab75310ab817","Type":"ContainerDied","Data":"6064086e636f877f09385ea527e86b97a1ebca226f1c8961cb9b93c7879ca930"} Oct 31 03:23:47.785240 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.785018 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6064086e636f877f09385ea527e86b97a1ebca226f1c8961cb9b93c7879ca930" Oct 31 03:23:47.809354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.809326 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx82p\" (UniqueName: \"kubernetes.io/projected/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-kube-api-access-dx82p\") pod \"maas-api-revoke-sub-c71d7383-d9lb4\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:47.809494 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.809369 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c71d7383-d9lb4\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:47.809494 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.809465 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c71d7383-d9lb4\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:47.911176 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.911142 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx82p\" (UniqueName: \"kubernetes.io/projected/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-kube-api-access-dx82p\") pod \"maas-api-revoke-sub-c71d7383-d9lb4\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:47.911381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.911189 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c71d7383-d9lb4\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:47.911381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.911222 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c71d7383-d9lb4\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:47.911963 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.911933 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c71d7383-d9lb4\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:47.913547 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.913529 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c71d7383-d9lb4\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:47.929872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:47.929848 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx82p\" (UniqueName: \"kubernetes.io/projected/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-kube-api-access-dx82p\") pod \"maas-api-revoke-sub-c71d7383-d9lb4\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:48.055260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:48.055183 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:48.392445 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:23:48.392409 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966ecee2_b7f6_4bb6_9f05_6af9a19b7274.slice/crio-7402ed140a5abdd479e4adbea369d3d673943abeee6d2b8bc33238df923a053e WatchSource:0}: Error finding container 7402ed140a5abdd479e4adbea369d3d673943abeee6d2b8bc33238df923a053e: Status 404 returned error can't find the container with id 7402ed140a5abdd479e4adbea369d3d673943abeee6d2b8bc33238df923a053e Oct 31 03:23:48.394417 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:48.394388 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4"] Oct 31 03:23:48.795726 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:48.795684 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" event={"ID":"966ecee2-b7f6-4bb6-9f05-6af9a19b7274","Type":"ContainerStarted","Data":"d95ae78ceba458726247955a387c745f16e31dce6a03bc26f4f41bd53b5b98c5"} Oct 31 03:23:48.795968 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:48.795747 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" event={"ID":"966ecee2-b7f6-4bb6-9f05-6af9a19b7274","Type":"ContainerStarted","Data":"7402ed140a5abdd479e4adbea369d3d673943abeee6d2b8bc33238df923a053e"} Oct 31 03:23:48.822772 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:48.822709 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" podStartSLOduration=1.822690648 podStartE2EDuration="1.822690648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:48.818128916 +0000 UTC m=+1710.498655044" watchObservedRunningTime="2026-10-31 03:23:48.822690648 +0000 UTC m=+1710.503216782" Oct 31 03:23:49.809655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:49.809620 4456 generic.go:356] "Generic (PLEG): container finished" podID="966ecee2-b7f6-4bb6-9f05-6af9a19b7274" containerID="d95ae78ceba458726247955a387c745f16e31dce6a03bc26f4f41bd53b5b98c5" exitCode=0 Oct 31 03:23:49.810097 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:49.809701 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" event={"ID":"966ecee2-b7f6-4bb6-9f05-6af9a19b7274","Type":"ContainerDied","Data":"d95ae78ceba458726247955a387c745f16e31dce6a03bc26f4f41bd53b5b98c5"} Oct 31 03:23:50.970999 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:50.970960 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:51.063286 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.063251 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-xks-ca\") pod \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " Oct 31 03:23:51.063510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.063389 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-openshift-ca\") pod \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " Oct 31 03:23:51.063510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.063442 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-kube-api-access-dx82p\" (UniqueName: \"kubernetes.io/projected/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-kube-api-access-dx82p\") pod \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\" (UID: \"966ecee2-b7f6-4bb6-9f05-6af9a19b7274\") " Oct 31 03:23:51.063893 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.063867 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-openshift-ca" pod "966ecee2-b7f6-4bb6-9f05-6af9a19b7274" (UID: "966ecee2-b7f6-4bb6-9f05-6af9a19b7274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:51.064196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.064147 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:51.068496 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.068398 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-xks-ca" pod "966ecee2-b7f6-4bb6-9f05-6af9a19b7274" (UID: "966ecee2-b7f6-4bb6-9f05-6af9a19b7274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:51.068496 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.068452 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-kube-api-access-dx82p" pod "966ecee2-b7f6-4bb6-9f05-6af9a19b7274" (UID: "966ecee2-b7f6-4bb6-9f05-6af9a19b7274"). InnerVolumeSpecName "kube-api-access-dx82p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:51.165432 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.165390 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx82p\" (UniqueName: \"kubernetes.io/projected/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-kube-api-access-dx82p\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:51.165432 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.165424 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966ecee2-b7f6-4bb6-9f05-6af9a19b7274-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:51.829396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.829364 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" Oct 31 03:23:51.829580 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.829398 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4" event={"ID":"966ecee2-b7f6-4bb6-9f05-6af9a19b7274","Type":"ContainerDied","Data":"7402ed140a5abdd479e4adbea369d3d673943abeee6d2b8bc33238df923a053e"} Oct 31 03:23:51.829580 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:51.829450 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7402ed140a5abdd479e4adbea369d3d673943abeee6d2b8bc33238df923a053e" Oct 31 03:23:52.496770 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:52.496735 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g"] Oct 31 03:23:52.497340 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:52.496956 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" podUID="90490dc2-8174-4f93-b613-db807dd23c34" containerName="maas-api" containerID="cri-o://e459d5521a99b8349e2bd684bca8af35a89a1f88326ab56688d2876c4fd42069" gracePeriod=30 Oct 31 03:23:52.900692 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:52.900591 4456 generic.go:356] "Generic (PLEG): container finished" podID="90490dc2-8174-4f93-b613-db807dd23c34" containerID="e459d5521a99b8349e2bd684bca8af35a89a1f88326ab56688d2876c4fd42069" exitCode=0 Oct 31 03:23:52.901297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:52.901169 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" event={"ID":"90490dc2-8174-4f93-b613-db807dd23c34","Type":"ContainerDied","Data":"e459d5521a99b8349e2bd684bca8af35a89a1f88326ab56688d2876c4fd42069"} Oct 31 03:23:52.958397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:52.958357 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k"] Oct 31 03:23:52.966221 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:52.966139 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-85cb59-hs84k"] Oct 31 03:23:52.996206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:52.996174 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:23:53.102720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.102684 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-metrics-certs\") pod \"90490dc2-8174-4f93-b613-db807dd23c34\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " Oct 31 03:23:53.102926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.102759 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90490dc2-8174-4f93-b613-db807dd23c34-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90490dc2-8174-4f93-b613-db807dd23c34-openshift-service-ca\") pod \"90490dc2-8174-4f93-b613-db807dd23c34\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " Oct 31 03:23:53.102926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.102843 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90490dc2-8174-4f93-b613-db807dd23c34-kube-api-access-qszf2\" (UniqueName: \"kubernetes.io/projected/90490dc2-8174-4f93-b613-db807dd23c34-kube-api-access-qszf2\") pod \"90490dc2-8174-4f93-b613-db807dd23c34\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " Oct 31 03:23:53.102926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.102867 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-maas-api-tls\") pod \"90490dc2-8174-4f93-b613-db807dd23c34\" (UID: \"90490dc2-8174-4f93-b613-db807dd23c34\") " Oct 31 03:23:53.103208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.103180 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90490dc2-8174-4f93-b613-db807dd23c34-openshift-service-ca" pod "90490dc2-8174-4f93-b613-db807dd23c34" (UID: "90490dc2-8174-4f93-b613-db807dd23c34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:53.103317 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.103300 4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90490dc2-8174-4f93-b613-db807dd23c34-openshift-service-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:53.105210 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.105172 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-metrics-certs" pod "90490dc2-8174-4f93-b613-db807dd23c34" (UID: "90490dc2-8174-4f93-b613-db807dd23c34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:53.105555 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.105526 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-maas-api-tls" pod "90490dc2-8174-4f93-b613-db807dd23c34" (UID: "90490dc2-8174-4f93-b613-db807dd23c34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:53.105671 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.105620 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90490dc2-8174-4f93-b613-db807dd23c34-kube-api-access-qszf2" pod "90490dc2-8174-4f93-b613-db807dd23c34" (UID: "90490dc2-8174-4f93-b613-db807dd23c34"). InnerVolumeSpecName "kube-api-access-qszf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:53.205075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.205008 4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-metrics-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:53.205075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.205076 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qszf2\" (UniqueName: \"kubernetes.io/projected/90490dc2-8174-4f93-b613-db807dd23c34-kube-api-access-qszf2\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:53.205301 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.205090 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90490dc2-8174-4f93-b613-db807dd23c34-maas-api-tls\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:53.709064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.708995 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx"] Oct 31 03:23:53.921342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.921304 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" event={"ID":"90490dc2-8174-4f93-b613-db807dd23c34","Type":"ContainerDied","Data":"f7e28de381d5937c4267a65b364dec338286e3bac445ab9c538f1aa862c07b88"} Oct 31 03:23:53.921623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.921603 4456 scope.go:122] "RemoveContainer" containerID="e459d5521a99b8349e2bd684bca8af35a89a1f88326ab56688d2876c4fd42069" Oct 31 03:23:53.921935 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.921916 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g" Oct 31 03:23:53.978119 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.975578 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g"] Oct 31 03:23:53.993482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.990752 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j"] Oct 31 03:23:53.993482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.993306 4456 cpu_manager.go:414] "containerMap: removing container" podUID="966ecee2-b7f6-4bb6-9f05-6af9a19b7274" containerName="revoke-keys" Oct 31 03:23:53.993482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.993331 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966ecee2-b7f6-4bb6-9f05-6af9a19b7274" containerName="revoke-keys" Oct 31 03:23:53.993482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.993379 4456 cpu_manager.go:414] "containerMap: removing container" podUID="90490dc2-8174-4f93-b613-db807dd23c34" containerName="maas-api" Oct 31 03:23:53.993482 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.993388 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90490dc2-8174-4f93-b613-db807dd23c34" containerName="maas-api" Oct 31 03:23:53.993948 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.993634 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="966ecee2-b7f6-4bb6-9f05-6af9a19b7274" containerName="revoke-keys" Oct 31 03:23:53.993948 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:53.993653 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="90490dc2-8174-4f93-b613-db807dd23c34" containerName="maas-api" Oct 31 03:23:54.002303 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.002270 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-85cb59-576bfdc8f9-r662g"] Oct 31 03:23:54.002486 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.002313 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j"] Oct 31 03:23:54.002486 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.002420 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.011145 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.010416 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:23:54.011297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.011241 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:23:54.120841 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.120787 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46zr\" (UniqueName: \"kubernetes.io/projected/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-kube-api-access-v46zr\") pod \"maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.121023 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.120890 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.121113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.121079 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.222100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.222062 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46zr\" (UniqueName: \"kubernetes.io/projected/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-kube-api-access-v46zr\") pod \"maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.222280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.222137 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.222280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.222254 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.222852 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.222827 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.224749 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.224721 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.237129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.236919 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46zr\" (UniqueName: \"kubernetes.io/projected/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-kube-api-access-v46zr\") pod \"maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.322480 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.322445 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:54.438432 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.438402 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8084c8fb-e70e-4476-b436-fd7173ac0e74" path="/var/lib/kubelet/pods/8084c8fb-e70e-4476-b436-fd7173ac0e74/volumes" Oct 31 03:23:54.438921 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.438905 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90490dc2-8174-4f93-b613-db807dd23c34" path="/var/lib/kubelet/pods/90490dc2-8174-4f93-b613-db807dd23c34/volumes" Oct 31 03:23:54.610673 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:23:54.610637 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c070c3_ba1a_4728_9b2e_cfc2f0ba9698.slice/crio-48b3e4ef9557a59cc0175a4ede33f62277273f89bb8cceadff2734ad707da028 WatchSource:0}: Error finding container 48b3e4ef9557a59cc0175a4ede33f62277273f89bb8cceadff2734ad707da028: Status 404 returned error can't find the container with id 48b3e4ef9557a59cc0175a4ede33f62277273f89bb8cceadff2734ad707da028 Oct 31 03:23:54.612261 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.612211 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j"] Oct 31 03:23:54.931606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:54.931574 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" event={"ID":"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698","Type":"ContainerStarted","Data":"48b3e4ef9557a59cc0175a4ede33f62277273f89bb8cceadff2734ad707da028"} Oct 31 03:23:55.944791 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:55.944761 4456 generic.go:356] "Generic (PLEG): container finished" podID="c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698" containerID="bcce3c1f2732183be98d097c0a8c193e05539dc5a55662c3bf9ac0cfb02d6266" exitCode=0 Oct 31 03:23:55.945234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:55.944848 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" event={"ID":"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698","Type":"ContainerDied","Data":"bcce3c1f2732183be98d097c0a8c193e05539dc5a55662c3bf9ac0cfb02d6266"} Oct 31 03:23:57.146590 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.146564 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:57.263811 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.263716 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-openshift-ca\") pod \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " Oct 31 03:23:57.263811 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.263806 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-xks-ca\") pod \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " Oct 31 03:23:57.264081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.263956 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-kube-api-access-v46zr\" (UniqueName: \"kubernetes.io/projected/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-kube-api-access-v46zr\") pod \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\" (UID: \"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698\") " Oct 31 03:23:57.264347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.264314 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-openshift-ca" pod "c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698" (UID: "c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:57.264559 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.264535 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:57.266657 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.266628 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-xks-ca" pod "c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698" (UID: "c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:57.266798 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.266777 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-kube-api-access-v46zr" pod "c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698" (UID: "c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698"). InnerVolumeSpecName "kube-api-access-v46zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:57.365681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.365642 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:57.365681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.365675 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v46zr\" (UniqueName: \"kubernetes.io/projected/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698-kube-api-access-v46zr\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:57.427403 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.427352 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" podUID="4d954ed3-9585-4249-8c87-32ee474126a7" containerName="main" containerID="cri-o://e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4" gracePeriod=45 Oct 31 03:23:57.695420 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.695393 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:23:57.768978 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.768937 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-model-cache\") pod \"4d954ed3-9585-4249-8c87-32ee474126a7\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " Oct 31 03:23:57.769179 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.769096 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d954ed3-9585-4249-8c87-32ee474126a7-kube-api-access-8vvpl\" (UniqueName: \"kubernetes.io/projected/4d954ed3-9585-4249-8c87-32ee474126a7-kube-api-access-8vvpl\") pod \"4d954ed3-9585-4249-8c87-32ee474126a7\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " Oct 31 03:23:57.769179 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.769172 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d954ed3-9585-4249-8c87-32ee474126a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d954ed3-9585-4249-8c87-32ee474126a7-tls-certs\") pod \"4d954ed3-9585-4249-8c87-32ee474126a7\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " Oct 31 03:23:57.769331 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.769210 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-dshm\") pod \"4d954ed3-9585-4249-8c87-32ee474126a7\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " Oct 31 03:23:57.769331 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.769238 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-model-cache" pod "4d954ed3-9585-4249-8c87-32ee474126a7" (UID: "4d954ed3-9585-4249-8c87-32ee474126a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:57.769493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.769340 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-tmp-dir\") pod \"4d954ed3-9585-4249-8c87-32ee474126a7\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " Oct 31 03:23:57.769493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.769381 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-home\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-home\") pod \"4d954ed3-9585-4249-8c87-32ee474126a7\" (UID: \"4d954ed3-9585-4249-8c87-32ee474126a7\") " Oct 31 03:23:57.769814 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.769790 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-tmp-dir" pod "4d954ed3-9585-4249-8c87-32ee474126a7" (UID: "4d954ed3-9585-4249-8c87-32ee474126a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:57.770003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.769950 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:57.770126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.770009 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:57.771219 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.771192 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-home" pod "4d954ed3-9585-4249-8c87-32ee474126a7" (UID: "4d954ed3-9585-4249-8c87-32ee474126a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:57.773178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.773153 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d954ed3-9585-4249-8c87-32ee474126a7-kube-api-access-8vvpl" pod "4d954ed3-9585-4249-8c87-32ee474126a7" (UID: "4d954ed3-9585-4249-8c87-32ee474126a7"). InnerVolumeSpecName "kube-api-access-8vvpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:57.773412 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.773394 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d954ed3-9585-4249-8c87-32ee474126a7-tls-certs" pod "4d954ed3-9585-4249-8c87-32ee474126a7" (UID: "4d954ed3-9585-4249-8c87-32ee474126a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:57.774283 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.774263 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-dshm" pod "4d954ed3-9585-4249-8c87-32ee474126a7" (UID: "4d954ed3-9585-4249-8c87-32ee474126a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:57.871858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.871766 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vvpl\" (UniqueName: \"kubernetes.io/projected/4d954ed3-9585-4249-8c87-32ee474126a7-kube-api-access-8vvpl\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:57.871858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.871803 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d954ed3-9585-4249-8c87-32ee474126a7-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:57.871858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.871816 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:57.871858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.871828 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d954ed3-9585-4249-8c87-32ee474126a7-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:57.975278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.975248 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" Oct 31 03:23:57.975492 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.975279 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j" event={"ID":"c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698","Type":"ContainerDied","Data":"48b3e4ef9557a59cc0175a4ede33f62277273f89bb8cceadff2734ad707da028"} Oct 31 03:23:57.975492 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.975328 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48b3e4ef9557a59cc0175a4ede33f62277273f89bb8cceadff2734ad707da028" Oct 31 03:23:57.977259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.977228 4456 generic.go:356] "Generic (PLEG): container finished" podID="4d954ed3-9585-4249-8c87-32ee474126a7" containerID="e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4" exitCode=0 Oct 31 03:23:57.977436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.977269 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" Oct 31 03:23:57.977436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.977297 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" event={"ID":"4d954ed3-9585-4249-8c87-32ee474126a7","Type":"ContainerDied","Data":"e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4"} Oct 31 03:23:57.977436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.977330 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v" event={"ID":"4d954ed3-9585-4249-8c87-32ee474126a7","Type":"ContainerDied","Data":"385dfbc22402a720c5713bcce74209846908b47b92bd03ac3a15eb94554735c6"} Oct 31 03:23:57.977436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.977351 4456 scope.go:122] "RemoveContainer" containerID="e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4" Oct 31 03:23:57.992108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.992018 4456 scope.go:122] "RemoveContainer" containerID="e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4" Oct 31 03:23:57.992431 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:23:57.992407 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4\": container with ID starting with e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4 not found: ID does not exist" containerID="e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4" Oct 31 03:23:57.992511 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:57.992436 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4"} err="failed to get container status \"e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4\": rpc error: code = NotFound desc = could not find container \"e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4\": container with ID starting with e0a197076cf5121186d011a23a79cfcd7b64e5ffcebe514b381abb564f1b71e4 not found: ID does not exist" Oct 31 03:23:58.010740 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:58.010715 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v"] Oct 31 03:23:58.020467 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:58.020438 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a2bec7/test-model-a2bec7-kserve-6f8d4bb747-jd27v"] Oct 31 03:23:58.436572 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:58.436536 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d954ed3-9585-4249-8c87-32ee474126a7" path="/var/lib/kubelet/pods/4d954ed3-9585-4249-8c87-32ee474126a7/volumes" Oct 31 03:23:59.087543 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.087512 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt"] Oct 31 03:23:59.588863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.588820 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk"] Oct 31 03:23:59.589743 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.589716 4456 cpu_manager.go:414] "containerMap: removing container" podUID="c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698" containerName="revoke-keys" Oct 31 03:23:59.589893 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.589874 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698" containerName="revoke-keys" Oct 31 03:23:59.590009 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.589935 4456 cpu_manager.go:414] "containerMap: removing container" podUID="4d954ed3-9585-4249-8c87-32ee474126a7" containerName="main" Oct 31 03:23:59.590009 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.589944 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d954ed3-9585-4249-8c87-32ee474126a7" containerName="main" Oct 31 03:23:59.590176 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.590166 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698" containerName="revoke-keys" Oct 31 03:23:59.590230 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.590188 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d954ed3-9585-4249-8c87-32ee474126a7" containerName="main" Oct 31 03:23:59.596286 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.595680 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:23:59.600836 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.600814 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:23:59.602846 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.602823 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:23:59.603241 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.603219 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk"] Oct 31 03:23:59.698162 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.698127 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e32820f-ddpsk\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:23:59.698344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.698168 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvkk\" (UniqueName: \"kubernetes.io/projected/97db43ad-85cf-42f7-93f2-f83364ecff3d-kube-api-access-6zvkk\") pod \"maas-api-revoke-sub-3e32820f-ddpsk\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:23:59.698344 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.698201 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e32820f-ddpsk\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:23:59.799280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.799241 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e32820f-ddpsk\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:23:59.799280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.799285 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvkk\" (UniqueName: \"kubernetes.io/projected/97db43ad-85cf-42f7-93f2-f83364ecff3d-kube-api-access-6zvkk\") pod \"maas-api-revoke-sub-3e32820f-ddpsk\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:23:59.799540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.799388 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e32820f-ddpsk\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:23:59.800261 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.800235 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e32820f-ddpsk\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:23:59.801866 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.801843 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e32820f-ddpsk\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:23:59.829848 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.829779 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvkk\" (UniqueName: \"kubernetes.io/projected/97db43ad-85cf-42f7-93f2-f83364ecff3d-kube-api-access-6zvkk\") pod \"maas-api-revoke-sub-3e32820f-ddpsk\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:23:59.922031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:23:59.921994 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:24:00.218120 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:24:00.218085 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97db43ad_85cf_42f7_93f2_f83364ecff3d.slice/crio-35c8caa5399d77466e57784de20e5cdae17127fd12aeea80f4db4e3efbe59de6 WatchSource:0}: Error finding container 35c8caa5399d77466e57784de20e5cdae17127fd12aeea80f4db4e3efbe59de6: Status 404 returned error can't find the container with id 35c8caa5399d77466e57784de20e5cdae17127fd12aeea80f4db4e3efbe59de6 Oct 31 03:24:00.218252 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:00.218157 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk"] Oct 31 03:24:01.018876 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:01.018839 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" event={"ID":"97db43ad-85cf-42f7-93f2-f83364ecff3d","Type":"ContainerStarted","Data":"8691a0c705ce73674ad3bd7fc19b849992279be0fd997c5a46d2f8d8d221bad4"} Oct 31 03:24:01.019272 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:01.018922 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" event={"ID":"97db43ad-85cf-42f7-93f2-f83364ecff3d","Type":"ContainerStarted","Data":"35c8caa5399d77466e57784de20e5cdae17127fd12aeea80f4db4e3efbe59de6"} Oct 31 03:24:01.043807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:01.043757 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" podStartSLOduration=2.043743072 podStartE2EDuration="2.043743072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:01.0422467 +0000 UTC m=+1722.722772826" watchObservedRunningTime="2026-10-31 03:24:01.043743072 +0000 UTC m=+1722.724269241" Oct 31 03:24:01.903154 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:01.903103 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs"] Oct 31 03:24:01.903625 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:01.903595 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" containerID="cri-o://989312fdbf5b95b023c1150e973e778863d119ee8708b246f10117cd4bcc034d" gracePeriod=30 Oct 31 03:24:02.159101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:02.157855 4456 generic.go:356] "Generic (PLEG): container finished" podID="97db43ad-85cf-42f7-93f2-f83364ecff3d" containerID="8691a0c705ce73674ad3bd7fc19b849992279be0fd997c5a46d2f8d8d221bad4" exitCode=0 Oct 31 03:24:02.159101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:02.157930 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" event={"ID":"97db43ad-85cf-42f7-93f2-f83364ecff3d","Type":"ContainerDied","Data":"8691a0c705ce73674ad3bd7fc19b849992279be0fd997c5a46d2f8d8d221bad4"} Oct 31 03:24:02.179366 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:02.179326 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:24:03.340219 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.340185 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:24:03.427887 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.427801 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-xks-ca\") pod \"97db43ad-85cf-42f7-93f2-f83364ecff3d\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " Oct 31 03:24:03.427887 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.427866 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97db43ad-85cf-42f7-93f2-f83364ecff3d-kube-api-access-6zvkk\" (UniqueName: \"kubernetes.io/projected/97db43ad-85cf-42f7-93f2-f83364ecff3d-kube-api-access-6zvkk\") pod \"97db43ad-85cf-42f7-93f2-f83364ecff3d\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " Oct 31 03:24:03.428150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.427927 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-openshift-ca\") pod \"97db43ad-85cf-42f7-93f2-f83364ecff3d\" (UID: \"97db43ad-85cf-42f7-93f2-f83364ecff3d\") " Oct 31 03:24:03.428347 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.428322 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-openshift-ca" pod "97db43ad-85cf-42f7-93f2-f83364ecff3d" (UID: "97db43ad-85cf-42f7-93f2-f83364ecff3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:03.430022 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.430001 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97db43ad-85cf-42f7-93f2-f83364ecff3d-kube-api-access-6zvkk" pod "97db43ad-85cf-42f7-93f2-f83364ecff3d" (UID: "97db43ad-85cf-42f7-93f2-f83364ecff3d"). InnerVolumeSpecName "kube-api-access-6zvkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:03.430127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.430073 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-xks-ca" pod "97db43ad-85cf-42f7-93f2-f83364ecff3d" (UID: "97db43ad-85cf-42f7-93f2-f83364ecff3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:03.529277 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.529235 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:03.529277 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.529270 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zvkk\" (UniqueName: \"kubernetes.io/projected/97db43ad-85cf-42f7-93f2-f83364ecff3d-kube-api-access-6zvkk\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:03.529508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.529284 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97db43ad-85cf-42f7-93f2-f83364ecff3d-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:03.620610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.620573 4456 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:24:03.620793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:03.620653 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:24:04.184440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:04.184199 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" Oct 31 03:24:04.184440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:04.184240 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk" event={"ID":"97db43ad-85cf-42f7-93f2-f83364ecff3d","Type":"ContainerDied","Data":"35c8caa5399d77466e57784de20e5cdae17127fd12aeea80f4db4e3efbe59de6"} Oct 31 03:24:04.184440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:04.184300 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35c8caa5399d77466e57784de20e5cdae17127fd12aeea80f4db4e3efbe59de6" Oct 31 03:24:07.269275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.268896 4456 generic.go:356] "Generic (PLEG): container finished" podID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerID="989312fdbf5b95b023c1150e973e778863d119ee8708b246f10117cd4bcc034d" exitCode=0 Oct 31 03:24:07.269275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.269141 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" event={"ID":"7a5e43a2-a261-4c0a-8ad6-34a5774497e4","Type":"ContainerDied","Data":"989312fdbf5b95b023c1150e973e778863d119ee8708b246f10117cd4bcc034d"} Oct 31 03:24:07.320731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.320651 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j"] Oct 31 03:24:07.323845 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.323817 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2bec7-j9d5j"] Oct 31 03:24:07.352874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.352280 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:24:07.430343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.430300 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-socket\") pod \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " Oct 31 03:24:07.430549 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.430358 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-certs\") pod \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " Oct 31 03:24:07.430549 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.430496 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-data\") pod \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " Oct 31 03:24:07.430675 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.430623 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-envoy\") pod \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " Oct 31 03:24:07.430731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.430680 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-podinfo\") pod \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " Oct 31 03:24:07.430731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.430712 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-token\" (UniqueName: \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-token\") pod \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " Oct 31 03:24:07.430853 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.430761 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-kube-api-access-4gzgr\" (UniqueName: \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-kube-api-access-4gzgr\") pod \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " Oct 31 03:24:07.430853 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.430793 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-credential-socket\") pod \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " Oct 31 03:24:07.430853 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.430843 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istiod-ca-cert\") pod \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\" (UID: \"7a5e43a2-a261-4c0a-8ad6-34a5774497e4\") " Oct 31 03:24:07.432412 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.432377 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-socket" pod "7a5e43a2-a261-4c0a-8ad6-34a5774497e4" (UID: "7a5e43a2-a261-4c0a-8ad6-34a5774497e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.432643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.432619 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-certs" pod "7a5e43a2-a261-4c0a-8ad6-34a5774497e4" (UID: "7a5e43a2-a261-4c0a-8ad6-34a5774497e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.432833 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.432813 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-data" pod "7a5e43a2-a261-4c0a-8ad6-34a5774497e4" (UID: "7a5e43a2-a261-4c0a-8ad6-34a5774497e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.436622 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.436591 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-credential-socket" pod "7a5e43a2-a261-4c0a-8ad6-34a5774497e4" (UID: "7a5e43a2-a261-4c0a-8ad6-34a5774497e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.437406 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.437367 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istiod-ca-cert" pod "7a5e43a2-a261-4c0a-8ad6-34a5774497e4" (UID: "7a5e43a2-a261-4c0a-8ad6-34a5774497e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:07.440545 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.440522 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-token" pod "7a5e43a2-a261-4c0a-8ad6-34a5774497e4" (UID: "7a5e43a2-a261-4c0a-8ad6-34a5774497e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:07.443124 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.442505 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-kube-api-access-4gzgr" pod "7a5e43a2-a261-4c0a-8ad6-34a5774497e4" (UID: "7a5e43a2-a261-4c0a-8ad6-34a5774497e4"). InnerVolumeSpecName "kube-api-access-4gzgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:07.444449 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.444424 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-envoy" pod "7a5e43a2-a261-4c0a-8ad6-34a5774497e4" (UID: "7a5e43a2-a261-4c0a-8ad6-34a5774497e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.445729 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.445708 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-podinfo" pod "7a5e43a2-a261-4c0a-8ad6-34a5774497e4" (UID: "7a5e43a2-a261-4c0a-8ad6-34a5774497e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:07.532531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.532454 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.532531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.532489 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.532531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.532504 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.532531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.532517 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.532531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.532530 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.532799 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.532545 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.532799 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.532558 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gzgr\" (UniqueName: \"kubernetes.io/projected/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-kube-api-access-4gzgr\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.532799 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.532574 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.532799 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:07.532589 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5e43a2-a261-4c0a-8ad6-34a5774497e4-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:08.282003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:08.281974 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" Oct 31 03:24:08.282493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:08.281974 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs" event={"ID":"7a5e43a2-a261-4c0a-8ad6-34a5774497e4","Type":"ContainerDied","Data":"a013d389e2ab91002584ac0be1dac28d87d5422137c603dc26a04637e2ec6035"} Oct 31 03:24:08.282493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:08.282142 4456 scope.go:122] "RemoveContainer" containerID="989312fdbf5b95b023c1150e973e778863d119ee8708b246f10117cd4bcc034d" Oct 31 03:24:08.314270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:08.314236 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs"] Oct 31 03:24:08.318189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:08.318162 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-7wwqs"] Oct 31 03:24:08.437264 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:08.437232 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" path="/var/lib/kubelet/pods/7a5e43a2-a261-4c0a-8ad6-34a5774497e4/volumes" Oct 31 03:24:08.437815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:08.437798 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698" path="/var/lib/kubelet/pods/c7c070c3-ba1a-4728-9b2e-cfc2f0ba9698/volumes" Oct 31 03:24:08.740056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:08.739979 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" podUID="da8a0ce3-d044-4698-9ec5-585412658ec1" containerName="main" containerID="cri-o://7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040" gracePeriod=45 Oct 31 03:24:09.028548 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.028524 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:24:09.153632 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.153597 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-home\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-home\") pod \"da8a0ce3-d044-4698-9ec5-585412658ec1\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " Oct 31 03:24:09.153831 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.153784 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-model-cache\") pod \"da8a0ce3-d044-4698-9ec5-585412658ec1\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " Oct 31 03:24:09.153903 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.153831 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8a0ce3-d044-4698-9ec5-585412658ec1-tls-certs\" (UniqueName: \"kubernetes.io/secret/da8a0ce3-d044-4698-9ec5-585412658ec1-tls-certs\") pod \"da8a0ce3-d044-4698-9ec5-585412658ec1\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " Oct 31 03:24:09.153903 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.153849 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-home" pod "da8a0ce3-d044-4698-9ec5-585412658ec1" (UID: "da8a0ce3-d044-4698-9ec5-585412658ec1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:09.153903 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.153861 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-dshm\") pod \"da8a0ce3-d044-4698-9ec5-585412658ec1\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " Oct 31 03:24:09.154109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.153921 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-tmp-dir\") pod \"da8a0ce3-d044-4698-9ec5-585412658ec1\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " Oct 31 03:24:09.154109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.153991 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8a0ce3-d044-4698-9ec5-585412658ec1-kube-api-access-ttll9\" (UniqueName: \"kubernetes.io/projected/da8a0ce3-d044-4698-9ec5-585412658ec1-kube-api-access-ttll9\") pod \"da8a0ce3-d044-4698-9ec5-585412658ec1\" (UID: \"da8a0ce3-d044-4698-9ec5-585412658ec1\") " Oct 31 03:24:09.154109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.153994 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-model-cache" pod "da8a0ce3-d044-4698-9ec5-585412658ec1" (UID: "da8a0ce3-d044-4698-9ec5-585412658ec1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:09.154402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.154381 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:09.154464 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.154408 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:09.154464 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.154434 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-tmp-dir" pod "da8a0ce3-d044-4698-9ec5-585412658ec1" (UID: "da8a0ce3-d044-4698-9ec5-585412658ec1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:09.156185 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.156153 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-dshm" pod "da8a0ce3-d044-4698-9ec5-585412658ec1" (UID: "da8a0ce3-d044-4698-9ec5-585412658ec1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:09.156658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.156639 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8a0ce3-d044-4698-9ec5-585412658ec1-tls-certs" pod "da8a0ce3-d044-4698-9ec5-585412658ec1" (UID: "da8a0ce3-d044-4698-9ec5-585412658ec1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:09.156741 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.156665 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8a0ce3-d044-4698-9ec5-585412658ec1-kube-api-access-ttll9" pod "da8a0ce3-d044-4698-9ec5-585412658ec1" (UID: "da8a0ce3-d044-4698-9ec5-585412658ec1"). InnerVolumeSpecName "kube-api-access-ttll9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:09.256072 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.255977 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8a0ce3-d044-4698-9ec5-585412658ec1-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:09.256072 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.256009 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:09.256072 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.256024 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8a0ce3-d044-4698-9ec5-585412658ec1-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:09.256072 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.256051 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttll9\" (UniqueName: \"kubernetes.io/projected/da8a0ce3-d044-4698-9ec5-585412658ec1-kube-api-access-ttll9\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:09.295846 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.295807 4456 generic.go:356] "Generic (PLEG): container finished" podID="da8a0ce3-d044-4698-9ec5-585412658ec1" containerID="7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040" exitCode=0 Oct 31 03:24:09.296287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.295865 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" Oct 31 03:24:09.296287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.295885 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" event={"ID":"da8a0ce3-d044-4698-9ec5-585412658ec1","Type":"ContainerDied","Data":"7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040"} Oct 31 03:24:09.296287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.295926 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx" event={"ID":"da8a0ce3-d044-4698-9ec5-585412658ec1","Type":"ContainerDied","Data":"67b2f6516c110f247fd0abaa19b86d8898ede4d139a6a8e931327f7ffd8dbf32"} Oct 31 03:24:09.296287 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.295943 4456 scope.go:122] "RemoveContainer" containerID="7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040" Oct 31 03:24:09.307894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.307875 4456 scope.go:122] "RemoveContainer" containerID="7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040" Oct 31 03:24:09.308195 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:09.308175 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040\": container with ID starting with 7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040 not found: ID does not exist" containerID="7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040" Oct 31 03:24:09.308266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.308205 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040"} err="failed to get container status \"7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040\": rpc error: code = NotFound desc = could not find container \"7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040\": container with ID starting with 7283d6a2f39f1669ee1456257a12bd65b22504781dd6ef47ef66549679b25040 not found: ID does not exist" Oct 31 03:24:09.324322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.324294 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx"] Oct 31 03:24:09.328189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:09.328168 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7269ae/test-model-7269ae-kserve-6d4bfcdc5-bthlx"] Oct 31 03:24:10.436254 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:10.436212 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8a0ce3-d044-4698-9ec5-585412658ec1" path="/var/lib/kubelet/pods/da8a0ce3-d044-4698-9ec5-585412658ec1/volumes" Oct 31 03:24:11.833493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:11.833458 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b"] Oct 31 03:24:12.120142 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.120051 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx"] Oct 31 03:24:12.120648 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.120631 4456 cpu_manager.go:414] "containerMap: removing container" podUID="da8a0ce3-d044-4698-9ec5-585412658ec1" containerName="main" Oct 31 03:24:12.120705 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.120648 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8a0ce3-d044-4698-9ec5-585412658ec1" containerName="main" Oct 31 03:24:12.120705 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.120674 4456 cpu_manager.go:414] "containerMap: removing container" podUID="97db43ad-85cf-42f7-93f2-f83364ecff3d" containerName="revoke-keys" Oct 31 03:24:12.120705 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.120679 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97db43ad-85cf-42f7-93f2-f83364ecff3d" containerName="revoke-keys" Oct 31 03:24:12.120705 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.120688 4456 cpu_manager.go:414] "containerMap: removing container" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" Oct 31 03:24:12.120705 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.120693 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" Oct 31 03:24:12.120914 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.120790 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8a0ce3-d044-4698-9ec5-585412658ec1" containerName="main" Oct 31 03:24:12.120914 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.120802 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="97db43ad-85cf-42f7-93f2-f83364ecff3d" containerName="revoke-keys" Oct 31 03:24:12.120914 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.120808 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5e43a2-a261-4c0a-8ad6-34a5774497e4" containerName="istio-proxy" Oct 31 03:24:12.124577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.124549 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.135375 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.132662 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:24:12.135375 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.132804 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:24:12.139778 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.138634 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx"] Oct 31 03:24:12.314267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.314106 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ps5\" (UniqueName: \"kubernetes.io/projected/6933b8e0-b326-4037-aae8-ba86176ee228-kube-api-access-k8ps5\") pod \"maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.314602 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.314564 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.318557 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.315077 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.419885 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.419844 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.420115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.420014 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ps5\" (UniqueName: \"kubernetes.io/projected/6933b8e0-b326-4037-aae8-ba86176ee228-kube-api-access-k8ps5\") pod \"maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.420195 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.420167 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.420599 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.420572 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.424741 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.423355 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.439751 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.439712 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ps5\" (UniqueName: \"kubernetes.io/projected/6933b8e0-b326-4037-aae8-ba86176ee228-kube-api-access-k8ps5\") pod \"maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.488483 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.488443 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:12.860117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:12.859211 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx"] Oct 31 03:24:12.919956 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:24:12.907006 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6933b8e0_b326_4037_aae8_ba86176ee228.slice/crio-47084b432fde3bfebe054f68aa8a1bf333e01d7637c5ac694ea61548385beffe WatchSource:0}: Error finding container 47084b432fde3bfebe054f68aa8a1bf333e01d7637c5ac694ea61548385beffe: Status 404 returned error can't find the container with id 47084b432fde3bfebe054f68aa8a1bf333e01d7637c5ac694ea61548385beffe Oct 31 03:24:13.373522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:13.373488 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" event={"ID":"6933b8e0-b326-4037-aae8-ba86176ee228","Type":"ContainerStarted","Data":"2a3104fef36841f2dbf4b328ab88159594679e1196e4b78f93a2a0f3f90a3286"} Oct 31 03:24:13.373673 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:13.373569 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" event={"ID":"6933b8e0-b326-4037-aae8-ba86176ee228","Type":"ContainerStarted","Data":"47084b432fde3bfebe054f68aa8a1bf333e01d7637c5ac694ea61548385beffe"} Oct 31 03:24:13.406089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:13.404096 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" podStartSLOduration=1.404075311 podStartE2EDuration="1.404075311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:13.403182561 +0000 UTC m=+1735.083708689" watchObservedRunningTime="2026-10-31 03:24:13.404075311 +0000 UTC m=+1735.084601432" Oct 31 03:24:14.119818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.119697 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" podUID="5b473df1-9dae-4fcf-989e-86597c522258" containerName="main" containerID="cri-o://42474901677c078cc046f756302a62f162fc7d504d78f00ff2e6672a0c2faea3" gracePeriod=45 Oct 31 03:24:14.385895 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.385804 4456 generic.go:356] "Generic (PLEG): container finished" podID="5b473df1-9dae-4fcf-989e-86597c522258" containerID="42474901677c078cc046f756302a62f162fc7d504d78f00ff2e6672a0c2faea3" exitCode=0 Oct 31 03:24:14.386079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.385884 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" event={"ID":"5b473df1-9dae-4fcf-989e-86597c522258","Type":"ContainerDied","Data":"42474901677c078cc046f756302a62f162fc7d504d78f00ff2e6672a0c2faea3"} Oct 31 03:24:14.388112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.388084 4456 generic.go:356] "Generic (PLEG): container finished" podID="6933b8e0-b326-4037-aae8-ba86176ee228" containerID="2a3104fef36841f2dbf4b328ab88159594679e1196e4b78f93a2a0f3f90a3286" exitCode=0 Oct 31 03:24:14.388234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.388131 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" event={"ID":"6933b8e0-b326-4037-aae8-ba86176ee228","Type":"ContainerDied","Data":"2a3104fef36841f2dbf4b328ab88159594679e1196e4b78f93a2a0f3f90a3286"} Oct 31 03:24:14.428600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.428574 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:24:14.484982 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.484945 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-model-cache\") pod \"5b473df1-9dae-4fcf-989e-86597c522258\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " Oct 31 03:24:14.485295 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.485270 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-model-cache" pod "5b473df1-9dae-4fcf-989e-86597c522258" (UID: "5b473df1-9dae-4fcf-989e-86597c522258"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:14.485295 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.485286 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-tmp-dir" pod "5b473df1-9dae-4fcf-989e-86597c522258" (UID: "5b473df1-9dae-4fcf-989e-86597c522258"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:14.485454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.485027 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-tmp-dir\") pod \"5b473df1-9dae-4fcf-989e-86597c522258\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " Oct 31 03:24:14.485454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.485400 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b473df1-9dae-4fcf-989e-86597c522258-kube-api-access-xh9bd\" (UniqueName: \"kubernetes.io/projected/5b473df1-9dae-4fcf-989e-86597c522258-kube-api-access-xh9bd\") pod \"5b473df1-9dae-4fcf-989e-86597c522258\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " Oct 31 03:24:14.485883 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.485862 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b473df1-9dae-4fcf-989e-86597c522258-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b473df1-9dae-4fcf-989e-86597c522258-tls-certs\") pod \"5b473df1-9dae-4fcf-989e-86597c522258\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " Oct 31 03:24:14.485971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.485924 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-home\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-home\") pod \"5b473df1-9dae-4fcf-989e-86597c522258\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " Oct 31 03:24:14.486322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.486297 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-home" pod "5b473df1-9dae-4fcf-989e-86597c522258" (UID: "5b473df1-9dae-4fcf-989e-86597c522258"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:14.486694 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.486674 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-dshm\") pod \"5b473df1-9dae-4fcf-989e-86597c522258\" (UID: \"5b473df1-9dae-4fcf-989e-86597c522258\") " Oct 31 03:24:14.488579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.487440 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:14.488579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.487768 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:14.488579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.487788 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:14.488980 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.488957 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b473df1-9dae-4fcf-989e-86597c522258-kube-api-access-xh9bd" pod "5b473df1-9dae-4fcf-989e-86597c522258" (UID: "5b473df1-9dae-4fcf-989e-86597c522258"). InnerVolumeSpecName "kube-api-access-xh9bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:14.496849 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.496826 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b473df1-9dae-4fcf-989e-86597c522258-tls-certs" pod "5b473df1-9dae-4fcf-989e-86597c522258" (UID: "5b473df1-9dae-4fcf-989e-86597c522258"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:14.497966 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.497927 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-dshm" pod "5b473df1-9dae-4fcf-989e-86597c522258" (UID: "5b473df1-9dae-4fcf-989e-86597c522258"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:14.588892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.588854 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b473df1-9dae-4fcf-989e-86597c522258-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:14.588892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.588886 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh9bd\" (UniqueName: \"kubernetes.io/projected/5b473df1-9dae-4fcf-989e-86597c522258-kube-api-access-xh9bd\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:14.588892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:14.588899 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b473df1-9dae-4fcf-989e-86597c522258-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:15.400651 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.400620 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" Oct 31 03:24:15.401189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.400659 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt" event={"ID":"5b473df1-9dae-4fcf-989e-86597c522258","Type":"ContainerDied","Data":"33c34583f08a5daf9b938a71b4c037440566316b1cc6c74afe61a17caeec1329"} Oct 31 03:24:15.401189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.400732 4456 scope.go:122] "RemoveContainer" containerID="42474901677c078cc046f756302a62f162fc7d504d78f00ff2e6672a0c2faea3" Oct 31 03:24:15.430812 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.430785 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt"] Oct 31 03:24:15.434859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.434829 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-62bf00/test-model-62bf00-kserve-c7d74d4cf-tq8nt"] Oct 31 03:24:15.581115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.581088 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:15.703699 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.703657 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-openshift-ca\") pod \"6933b8e0-b326-4037-aae8-ba86176ee228\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " Oct 31 03:24:15.703862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.703737 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6933b8e0-b326-4037-aae8-ba86176ee228-kube-api-access-k8ps5\" (UniqueName: \"kubernetes.io/projected/6933b8e0-b326-4037-aae8-ba86176ee228-kube-api-access-k8ps5\") pod \"6933b8e0-b326-4037-aae8-ba86176ee228\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " Oct 31 03:24:15.703862 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.703782 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-xks-ca\") pod \"6933b8e0-b326-4037-aae8-ba86176ee228\" (UID: \"6933b8e0-b326-4037-aae8-ba86176ee228\") " Oct 31 03:24:15.704168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.704143 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-openshift-ca" pod "6933b8e0-b326-4037-aae8-ba86176ee228" (UID: "6933b8e0-b326-4037-aae8-ba86176ee228"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:15.704329 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.704315 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:15.705878 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.705854 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-xks-ca" pod "6933b8e0-b326-4037-aae8-ba86176ee228" (UID: "6933b8e0-b326-4037-aae8-ba86176ee228"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:15.705953 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.705938 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6933b8e0-b326-4037-aae8-ba86176ee228-kube-api-access-k8ps5" pod "6933b8e0-b326-4037-aae8-ba86176ee228" (UID: "6933b8e0-b326-4037-aae8-ba86176ee228"). InnerVolumeSpecName "kube-api-access-k8ps5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:15.808086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.805997 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ps5\" (UniqueName: \"kubernetes.io/projected/6933b8e0-b326-4037-aae8-ba86176ee228-kube-api-access-k8ps5\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:15.808086 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:15.806053 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6933b8e0-b326-4037-aae8-ba86176ee228-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:16.318981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.318941 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp"] Oct 31 03:24:16.319686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.319666 4456 cpu_manager.go:414] "containerMap: removing container" podUID="5b473df1-9dae-4fcf-989e-86597c522258" containerName="main" Oct 31 03:24:16.319686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.319685 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b473df1-9dae-4fcf-989e-86597c522258" containerName="main" Oct 31 03:24:16.319857 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.319697 4456 cpu_manager.go:414] "containerMap: removing container" podUID="6933b8e0-b326-4037-aae8-ba86176ee228" containerName="revoke-keys" Oct 31 03:24:16.319857 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.319702 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6933b8e0-b326-4037-aae8-ba86176ee228" containerName="revoke-keys" Oct 31 03:24:16.319857 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.319823 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b473df1-9dae-4fcf-989e-86597c522258" containerName="main" Oct 31 03:24:16.319857 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.319832 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6933b8e0-b326-4037-aae8-ba86176ee228" containerName="revoke-keys" Oct 31 03:24:16.323378 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.323360 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.331675 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.331650 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp"] Oct 31 03:24:16.410512 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.410481 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" Oct 31 03:24:16.410952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.410517 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx" event={"ID":"6933b8e0-b326-4037-aae8-ba86176ee228","Type":"ContainerDied","Data":"47084b432fde3bfebe054f68aa8a1bf333e01d7637c5ac694ea61548385beffe"} Oct 31 03:24:16.410952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.410568 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47084b432fde3bfebe054f68aa8a1bf333e01d7637c5ac694ea61548385beffe" Oct 31 03:24:16.418328 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.418140 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqprg\" (UniqueName: \"kubernetes.io/projected/9911df50-1921-4238-8fb5-08d64698002c-kube-api-access-bqprg\") pod \"maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.418328 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.418196 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9911df50-1921-4238-8fb5-08d64698002c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.418463 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.418340 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9911df50-1921-4238-8fb5-08d64698002c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.436321 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.436295 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b473df1-9dae-4fcf-989e-86597c522258" path="/var/lib/kubelet/pods/5b473df1-9dae-4fcf-989e-86597c522258/volumes" Oct 31 03:24:16.519799 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.519759 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqprg\" (UniqueName: \"kubernetes.io/projected/9911df50-1921-4238-8fb5-08d64698002c-kube-api-access-bqprg\") pod \"maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.520010 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.519893 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9911df50-1921-4238-8fb5-08d64698002c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.520132 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.520024 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9911df50-1921-4238-8fb5-08d64698002c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.520704 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.520680 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9911df50-1921-4238-8fb5-08d64698002c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.522280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.522259 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9911df50-1921-4238-8fb5-08d64698002c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.532768 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.532747 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqprg\" (UniqueName: \"kubernetes.io/projected/9911df50-1921-4238-8fb5-08d64698002c-kube-api-access-bqprg\") pod \"maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.638626 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.638542 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:16.937652 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:24:16.937603 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9911df50_1921_4238_8fb5_08d64698002c.slice/crio-00ff9d4f97b96c285cfbf89272dd2f2862e58a05bab9e02039ce2b91ee4f7ca6 WatchSource:0}: Error finding container 00ff9d4f97b96c285cfbf89272dd2f2862e58a05bab9e02039ce2b91ee4f7ca6: Status 404 returned error can't find the container with id 00ff9d4f97b96c285cfbf89272dd2f2862e58a05bab9e02039ce2b91ee4f7ca6 Oct 31 03:24:16.942402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:16.942367 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp"] Oct 31 03:24:17.423395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:17.423352 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" event={"ID":"9911df50-1921-4238-8fb5-08d64698002c","Type":"ContainerStarted","Data":"e2142176473fd99d5911132016eed25496a72e70c2e4ac96f66ad7facf12ab1d"} Oct 31 03:24:17.423778 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:17.423413 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" event={"ID":"9911df50-1921-4238-8fb5-08d64698002c","Type":"ContainerStarted","Data":"00ff9d4f97b96c285cfbf89272dd2f2862e58a05bab9e02039ce2b91ee4f7ca6"} Oct 31 03:24:17.444988 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:17.444934 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" podStartSLOduration=1.4449205410000001 podStartE2EDuration="1.444920541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:17.444370186 +0000 UTC m=+1739.124896311" watchObservedRunningTime="2026-10-31 03:24:17.444920541 +0000 UTC m=+1739.125446667" Oct 31 03:24:18.471200 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:18.471160 4456 generic.go:356] "Generic (PLEG): container finished" podID="9911df50-1921-4238-8fb5-08d64698002c" containerID="e2142176473fd99d5911132016eed25496a72e70c2e4ac96f66ad7facf12ab1d" exitCode=0 Oct 31 03:24:18.471642 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:18.471253 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" event={"ID":"9911df50-1921-4238-8fb5-08d64698002c","Type":"ContainerDied","Data":"e2142176473fd99d5911132016eed25496a72e70c2e4ac96f66ad7facf12ab1d"} Oct 31 03:24:19.587895 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.587860 4456 scope.go:122] "RemoveContainer" containerID="a5f26be4a85941dbf689c835f919cac6a26f88685219e1abb1cbedb7fa2d0a04" Oct 31 03:24:19.600436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.600409 4456 scope.go:122] "RemoveContainer" containerID="06522683fad1101988bc7f0af015ad25310b4068a3aa379127dc7585b7507a78" Oct 31 03:24:19.612667 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.612644 4456 scope.go:122] "RemoveContainer" containerID="ba23a9ed464c5afd7da3494849804e7936a5ed14cf73aa6d4cade2c703d2e806" Oct 31 03:24:19.623029 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.623010 4456 scope.go:122] "RemoveContainer" containerID="c43d07c9036a09517a20ffd4c97ee00bf835716125f843f0e356f52c0785773e" Oct 31 03:24:19.652797 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.652776 4456 scope.go:122] "RemoveContainer" containerID="6f9698c509eb86099670284bcb46a704fec4c1f25ebc17cc57ef47c50a30e495" Oct 31 03:24:19.653866 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.653845 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:19.779078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.779048 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9911df50-1921-4238-8fb5-08d64698002c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9911df50-1921-4238-8fb5-08d64698002c-maas-api-openshift-ca\") pod \"9911df50-1921-4238-8fb5-08d64698002c\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " Oct 31 03:24:19.779212 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.779109 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9911df50-1921-4238-8fb5-08d64698002c-kube-api-access-bqprg\" (UniqueName: \"kubernetes.io/projected/9911df50-1921-4238-8fb5-08d64698002c-kube-api-access-bqprg\") pod \"9911df50-1921-4238-8fb5-08d64698002c\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " Oct 31 03:24:19.779212 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.779143 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9911df50-1921-4238-8fb5-08d64698002c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9911df50-1921-4238-8fb5-08d64698002c-maas-api-xks-ca\") pod \"9911df50-1921-4238-8fb5-08d64698002c\" (UID: \"9911df50-1921-4238-8fb5-08d64698002c\") " Oct 31 03:24:19.779376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.779354 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9911df50-1921-4238-8fb5-08d64698002c-maas-api-openshift-ca" pod "9911df50-1921-4238-8fb5-08d64698002c" (UID: "9911df50-1921-4238-8fb5-08d64698002c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:19.779605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.779584 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9911df50-1921-4238-8fb5-08d64698002c-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:19.781211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.781183 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9911df50-1921-4238-8fb5-08d64698002c-kube-api-access-bqprg" pod "9911df50-1921-4238-8fb5-08d64698002c" (UID: "9911df50-1921-4238-8fb5-08d64698002c"). InnerVolumeSpecName "kube-api-access-bqprg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:19.781313 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.781260 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9911df50-1921-4238-8fb5-08d64698002c-maas-api-xks-ca" pod "9911df50-1921-4238-8fb5-08d64698002c" (UID: "9911df50-1921-4238-8fb5-08d64698002c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:19.881411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.881372 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqprg\" (UniqueName: \"kubernetes.io/projected/9911df50-1921-4238-8fb5-08d64698002c-kube-api-access-bqprg\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:19.881411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:19.881406 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9911df50-1921-4238-8fb5-08d64698002c-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:20.491491 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:20.491456 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" Oct 31 03:24:20.491491 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:20.491480 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp" event={"ID":"9911df50-1921-4238-8fb5-08d64698002c","Type":"ContainerDied","Data":"00ff9d4f97b96c285cfbf89272dd2f2862e58a05bab9e02039ce2b91ee4f7ca6"} Oct 31 03:24:20.491712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:20.491541 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00ff9d4f97b96c285cfbf89272dd2f2862e58a05bab9e02039ce2b91ee4f7ca6" Oct 31 03:24:26.468573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:26.468537 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv"] Oct 31 03:24:26.590063 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:26.589200 4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-7269ae: configmap "payload-processing-plugins-e2e-inf-b-7269ae" not found Oct 31 03:24:26.590063 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:26.589282 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume podName:ae6d2b26-4cc9-4af1-b951-aa1204ea91fb nodeName:}" failed. No retries permitted until 2026-10-31 03:24:27.089261089 +0000 UTC m=+1748.769787198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume") pod "payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" (UID: "ae6d2b26-4cc9-4af1-b951-aa1204ea91fb") : configmap "payload-processing-plugins-e2e-inf-b-7269ae" not found Oct 31 03:24:26.627819 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:26.627779 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx"] Oct 31 03:24:26.644396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:26.644353 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62bf00-4gznx"] Oct 31 03:24:26.866952 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:26.866808 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" podUID="814edc80-f564-47c1-a4e5-dc99d746e038" containerName="main" containerID="cri-o://00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7" gracePeriod=45 Oct 31 03:24:26.985773 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:26.985729 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" podUID="814edc80-f564-47c1-a4e5-dc99d746e038" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": dial tcp 10.133.0.155:8000: connect: connection refused" Oct 31 03:24:27.099164 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:27.099116 4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-7269ae: configmap "payload-processing-plugins-e2e-inf-b-7269ae" not found Oct 31 03:24:27.099958 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:27.099544 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume podName:ae6d2b26-4cc9-4af1-b951-aa1204ea91fb nodeName:}" failed. No retries permitted until 2026-10-31 03:24:28.099515163 +0000 UTC m=+1749.780041272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume") pod "payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" (UID: "ae6d2b26-4cc9-4af1-b951-aa1204ea91fb") : configmap "payload-processing-plugins-e2e-inf-b-7269ae" not found Oct 31 03:24:27.166200 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.166177 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:24:27.199944 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.199902 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/814edc80-f564-47c1-a4e5-dc99d746e038-tls-certs\" (UniqueName: \"kubernetes.io/secret/814edc80-f564-47c1-a4e5-dc99d746e038-tls-certs\") pod \"814edc80-f564-47c1-a4e5-dc99d746e038\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " Oct 31 03:24:27.200168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.199971 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-model-cache\") pod \"814edc80-f564-47c1-a4e5-dc99d746e038\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " Oct 31 03:24:27.200168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.200068 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-dshm\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-dshm\") pod \"814edc80-f564-47c1-a4e5-dc99d746e038\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " Oct 31 03:24:27.200168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.200126 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814edc80-f564-47c1-a4e5-dc99d746e038-kube-api-access-9th27\" (UniqueName: \"kubernetes.io/projected/814edc80-f564-47c1-a4e5-dc99d746e038-kube-api-access-9th27\") pod \"814edc80-f564-47c1-a4e5-dc99d746e038\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " Oct 31 03:24:27.200380 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.200187 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-tmp-dir\") pod \"814edc80-f564-47c1-a4e5-dc99d746e038\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " Oct 31 03:24:27.200380 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.200310 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-model-cache" pod "814edc80-f564-47c1-a4e5-dc99d746e038" (UID: "814edc80-f564-47c1-a4e5-dc99d746e038"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:27.200500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.200441 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-home\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-home\") pod \"814edc80-f564-47c1-a4e5-dc99d746e038\" (UID: \"814edc80-f564-47c1-a4e5-dc99d746e038\") " Oct 31 03:24:27.200709 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.200679 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-tmp-dir" pod "814edc80-f564-47c1-a4e5-dc99d746e038" (UID: "814edc80-f564-47c1-a4e5-dc99d746e038"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:27.201160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.200865 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-home" pod "814edc80-f564-47c1-a4e5-dc99d746e038" (UID: "814edc80-f564-47c1-a4e5-dc99d746e038"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:27.201160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.201099 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:27.201160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.201120 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:27.201160 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.201135 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:27.203745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.203717 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814edc80-f564-47c1-a4e5-dc99d746e038-kube-api-access-9th27" pod "814edc80-f564-47c1-a4e5-dc99d746e038" (UID: "814edc80-f564-47c1-a4e5-dc99d746e038"). InnerVolumeSpecName "kube-api-access-9th27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:27.205407 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.205386 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-dshm" pod "814edc80-f564-47c1-a4e5-dc99d746e038" (UID: "814edc80-f564-47c1-a4e5-dc99d746e038"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:27.205515 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.205405 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814edc80-f564-47c1-a4e5-dc99d746e038-tls-certs" pod "814edc80-f564-47c1-a4e5-dc99d746e038" (UID: "814edc80-f564-47c1-a4e5-dc99d746e038"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:27.302826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.302788 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814edc80-f564-47c1-a4e5-dc99d746e038-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:27.302826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.302824 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9th27\" (UniqueName: \"kubernetes.io/projected/814edc80-f564-47c1-a4e5-dc99d746e038-kube-api-access-9th27\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:27.303053 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.302839 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814edc80-f564-47c1-a4e5-dc99d746e038-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:27.629293 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.629256 4456 generic.go:356] "Generic (PLEG): container finished" podID="814edc80-f564-47c1-a4e5-dc99d746e038" containerID="00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7" exitCode=0 Oct 31 03:24:27.629820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.629447 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" event={"ID":"814edc80-f564-47c1-a4e5-dc99d746e038","Type":"ContainerDied","Data":"00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7"} Oct 31 03:24:27.629820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.629476 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" event={"ID":"814edc80-f564-47c1-a4e5-dc99d746e038","Type":"ContainerDied","Data":"818efd1544994fa96e95cc795d7c738c5bdf25e2bd3897c51c850d36b1dd80c8"} Oct 31 03:24:27.629820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.629492 4456 scope.go:122] "RemoveContainer" containerID="00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7" Oct 31 03:24:27.629820 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.629622 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b" Oct 31 03:24:27.670315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.669247 4456 scope.go:122] "RemoveContainer" containerID="00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7" Oct 31 03:24:27.671427 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:27.671396 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7\": container with ID starting with 00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7 not found: ID does not exist" containerID="00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7" Oct 31 03:24:27.671558 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.671436 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7"} err="failed to get container status \"00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7\": rpc error: code = NotFound desc = could not find container \"00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7\": container with ID starting with 00c3649e7009f5d1512e730628a8ddaea34fb87e9767048b698170c1dc25f5d7 not found: ID does not exist" Oct 31 03:24:27.675605 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.675582 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b"] Oct 31 03:24:27.679509 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.679487 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b79c13/test-model-b79c13-kserve-54d8d64bbf-dr42b"] Oct 31 03:24:27.719165 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.719130 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp"] Oct 31 03:24:27.723927 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:27.723891 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7269ae-c5jbp"] Oct 31 03:24:28.125144 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:28.125104 4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-7269ae: configmap "payload-processing-plugins-e2e-inf-b-7269ae" not found Oct 31 03:24:28.125313 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:28.125187 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume podName:ae6d2b26-4cc9-4af1-b951-aa1204ea91fb nodeName:}" failed. No retries permitted until 2026-10-31 03:24:30.125169666 +0000 UTC m=+1751.805695770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume") pod "payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" (UID: "ae6d2b26-4cc9-4af1-b951-aa1204ea91fb") : configmap "payload-processing-plugins-e2e-inf-b-7269ae" not found Oct 31 03:24:28.437490 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:28.437458 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6933b8e0-b326-4037-aae8-ba86176ee228" path="/var/lib/kubelet/pods/6933b8e0-b326-4037-aae8-ba86176ee228/volumes" Oct 31 03:24:28.437911 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:28.437896 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814edc80-f564-47c1-a4e5-dc99d746e038" path="/var/lib/kubelet/pods/814edc80-f564-47c1-a4e5-dc99d746e038/volumes" Oct 31 03:24:28.438309 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:28.438296 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9911df50-1921-4238-8fb5-08d64698002c" path="/var/lib/kubelet/pods/9911df50-1921-4238-8fb5-08d64698002c/volumes" Oct 31 03:24:30.152343 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:30.152303 4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-7269ae: configmap "payload-processing-plugins-e2e-inf-b-7269ae" not found Oct 31 03:24:30.152809 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:30.152400 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume podName:ae6d2b26-4cc9-4af1-b951-aa1204ea91fb nodeName:}" failed. No retries permitted until 2026-10-31 03:24:34.15237794 +0000 UTC m=+1755.832904060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume") pod "payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" (UID: "ae6d2b26-4cc9-4af1-b951-aa1204ea91fb") : configmap "payload-processing-plugins-e2e-inf-b-7269ae" not found Oct 31 03:24:31.581572 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:31.581433 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" podUID="ae6d2b26-4cc9-4af1-b951-aa1204ea91fb" containerName="payload-processing" containerID="cri-o://d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d" gracePeriod=25 Oct 31 03:24:31.859901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:31.859873 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:24:31.979948 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:31.979904 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-kube-api-access-wnhbw\" (UniqueName: \"kubernetes.io/projected/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-kube-api-access-wnhbw\") pod \"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb\" (UID: \"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb\") " Oct 31 03:24:31.980156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:31.980109 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume\") pod \"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb\" (UID: \"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb\") " Oct 31 03:24:31.980583 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:31.980549 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume" pod "ae6d2b26-4cc9-4af1-b951-aa1204ea91fb" (UID: "ae6d2b26-4cc9-4af1-b951-aa1204ea91fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:31.980779 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:31.980718 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:31.982463 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:31.982434 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-kube-api-access-wnhbw" pod "ae6d2b26-4cc9-4af1-b951-aa1204ea91fb" (UID: "ae6d2b26-4cc9-4af1-b951-aa1204ea91fb"). InnerVolumeSpecName "kube-api-access-wnhbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:32.082070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:32.082021 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnhbw\" (UniqueName: \"kubernetes.io/projected/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb-kube-api-access-wnhbw\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:32.141673 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:32.141589 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:24:32.698239 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:32.698201 4456 generic.go:356] "Generic (PLEG): container finished" podID="ae6d2b26-4cc9-4af1-b951-aa1204ea91fb" containerID="d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d" exitCode=0 Oct 31 03:24:32.698666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:32.698252 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" Oct 31 03:24:32.698666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:32.698276 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" event={"ID":"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb","Type":"ContainerDied","Data":"d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d"} Oct 31 03:24:32.698666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:32.698326 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv" event={"ID":"ae6d2b26-4cc9-4af1-b951-aa1204ea91fb","Type":"ContainerDied","Data":"909112f83202006b38080b56f504b0c8545b2c13540cbab271daa492be1d3f8c"} Oct 31 03:24:32.698666 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:32.698355 4456 scope.go:122] "RemoveContainer" containerID="d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d" Oct 31 03:24:32.712122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:32.712097 4456 scope.go:122] "RemoveContainer" containerID="d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d" Oct 31 03:24:32.712414 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:32.712395 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d\": container with ID starting with d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d not found: ID does not exist" containerID="d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d" Oct 31 03:24:32.712467 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:32.712421 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d"} err="failed to get container status \"d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d\": rpc error: code = NotFound desc = could not find container \"d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d\": container with ID starting with d3c499a18a16b7e77ecedacf24b7f621efd66c736c2ea5d7d15d02c45045289d not found: ID does not exist" Oct 31 03:24:32.722750 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:32.722724 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv"] Oct 31 03:24:32.730551 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:32.728451 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7269ae-97bc4ddb7-fxvmv"] Oct 31 03:24:34.435282 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:34.435254 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6d2b26-4cc9-4af1-b951-aa1204ea91fb" path="/var/lib/kubelet/pods/ae6d2b26-4cc9-4af1-b951-aa1204ea91fb/volumes" Oct 31 03:24:35.717141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:35.717107 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx"] Oct 31 03:24:35.717671 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:35.717388 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerName="istio-proxy" containerID="cri-o://aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4" gracePeriod=30 Oct 31 03:24:35.721590 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:35.721564 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9"] Oct 31 03:24:35.721775 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:35.721753 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" podUID="81d96e62-46ab-4df9-95b8-da022d4fce60" containerName="istio-proxy" containerID="cri-o://98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746" gracePeriod=30 Oct 31 03:24:35.740965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:35.740935 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp"] Oct 31 03:24:35.741140 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:35.741120 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerName="istio-proxy" containerID="cri-o://2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda" gracePeriod=30 Oct 31 03:24:36.241424 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.241387 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt"] Oct 31 03:24:36.242250 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.242219 4456 cpu_manager.go:414] "containerMap: removing container" podUID="ae6d2b26-4cc9-4af1-b951-aa1204ea91fb" containerName="payload-processing" Oct 31 03:24:36.242250 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.242248 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6d2b26-4cc9-4af1-b951-aa1204ea91fb" containerName="payload-processing" Oct 31 03:24:36.242508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.242281 4456 cpu_manager.go:414] "containerMap: removing container" podUID="814edc80-f564-47c1-a4e5-dc99d746e038" containerName="main" Oct 31 03:24:36.242508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.242289 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814edc80-f564-47c1-a4e5-dc99d746e038" containerName="main" Oct 31 03:24:36.242508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.242321 4456 cpu_manager.go:414] "containerMap: removing container" podUID="9911df50-1921-4238-8fb5-08d64698002c" containerName="revoke-keys" Oct 31 03:24:36.242508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.242330 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9911df50-1921-4238-8fb5-08d64698002c" containerName="revoke-keys" Oct 31 03:24:36.242508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.242480 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6d2b26-4cc9-4af1-b951-aa1204ea91fb" containerName="payload-processing" Oct 31 03:24:36.242508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.242502 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="814edc80-f564-47c1-a4e5-dc99d746e038" containerName="main" Oct 31 03:24:36.242947 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.242517 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9911df50-1921-4238-8fb5-08d64698002c" containerName="revoke-keys" Oct 31 03:24:36.263077 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.249876 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.263077 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.254731 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt"] Oct 31 03:24:36.343723 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.343690 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.343893 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.343737 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.343999 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.343975 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gbnz\" (UniqueName: \"kubernetes.io/projected/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-kube-api-access-9gbnz\") pod \"maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.345506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.345487 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:24:36.345731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.345714 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:24:36.445395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.445369 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gbnz\" (UniqueName: \"kubernetes.io/projected/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-kube-api-access-9gbnz\") pod \"maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.445557 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.445529 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.445657 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.445570 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.446205 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.446177 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.448457 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.448431 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.459563 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.459538 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gbnz\" (UniqueName: \"kubernetes.io/projected/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-kube-api-access-9gbnz\") pod \"maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.582253 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.582163 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:36.694674 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.693842 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld"] Oct 31 03:24:36.694674 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.694107 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerName="istio-proxy" containerID="cri-o://57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a" gracePeriod=30 Oct 31 03:24:36.702175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.699329 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q"] Oct 31 03:24:36.702175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.699577 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerName="istio-proxy" containerID="cri-o://4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8" gracePeriod=30 Oct 31 03:24:36.895967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:36.895935 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt"] Oct 31 03:24:37.407093 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.407008 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8"] Oct 31 03:24:37.411311 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.411294 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.419210 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.419181 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8"] Oct 31 03:24:37.560297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.560258 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.560567 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.560539 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsccr\" (UniqueName: \"kubernetes.io/projected/d1497ff1-e950-4e9e-bdd3-d8b80036744d-kube-api-access-vsccr\") pod \"maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.560665 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.560645 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.662647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.662542 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsccr\" (UniqueName: \"kubernetes.io/projected/d1497ff1-e950-4e9e-bdd3-d8b80036744d-kube-api-access-vsccr\") pod \"maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.662647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.662637 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.662841 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.662751 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.663622 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.663594 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.665263 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.665238 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.676984 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.676957 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsccr\" (UniqueName: \"kubernetes.io/projected/d1497ff1-e950-4e9e-bdd3-d8b80036744d-kube-api-access-vsccr\") pod \"maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.728561 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.728531 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:37.749121 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.749081 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" event={"ID":"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2","Type":"ContainerStarted","Data":"aaaad7701568d4daf7595503856446ec609927077172ed1528aa93c6010a82b0"} Oct 31 03:24:37.749254 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.749165 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" event={"ID":"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2","Type":"ContainerStarted","Data":"4a8f7a2796aa91320bb93194e1d81dddc3a6db8d8b64a9454b262b1e4556190a"} Oct 31 03:24:37.771717 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:37.771659 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" podStartSLOduration=1.771639132 podStartE2EDuration="1.771639132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:37.76896001 +0000 UTC m=+1759.449486141" watchObservedRunningTime="2026-10-31 03:24:37.771639132 +0000 UTC m=+1759.452165260" Oct 31 03:24:38.048664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:38.048581 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8"] Oct 31 03:24:38.052674 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:24:38.052641 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1497ff1_e950_4e9e_bdd3_d8b80036744d.slice/crio-4fe28b6c054cbbb105a22d3e3c33775667703dbc84195804ebe7a532c7705701 WatchSource:0}: Error finding container 4fe28b6c054cbbb105a22d3e3c33775667703dbc84195804ebe7a532c7705701: Status 404 returned error can't find the container with id 4fe28b6c054cbbb105a22d3e3c33775667703dbc84195804ebe7a532c7705701 Oct 31 03:24:38.761960 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:38.761918 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" event={"ID":"d1497ff1-e950-4e9e-bdd3-d8b80036744d","Type":"ContainerStarted","Data":"265db5e0d8ebbb50c5f7d411ff2f29a15cbc75b8bbbdabadee5282e2ac5dd923"} Oct 31 03:24:38.762189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:38.761989 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" event={"ID":"d1497ff1-e950-4e9e-bdd3-d8b80036744d","Type":"ContainerStarted","Data":"4fe28b6c054cbbb105a22d3e3c33775667703dbc84195804ebe7a532c7705701"} Oct 31 03:24:38.763703 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:38.763676 4456 generic.go:356] "Generic (PLEG): container finished" podID="4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2" containerID="aaaad7701568d4daf7595503856446ec609927077172ed1528aa93c6010a82b0" exitCode=0 Oct 31 03:24:38.763831 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:38.763723 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" event={"ID":"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2","Type":"ContainerDied","Data":"aaaad7701568d4daf7595503856446ec609927077172ed1528aa93c6010a82b0"} Oct 31 03:24:38.783850 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:38.783806 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" podStartSLOduration=1.783794947 podStartE2EDuration="1.783794947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:38.782708549 +0000 UTC m=+1760.463234678" watchObservedRunningTime="2026-10-31 03:24:38.783794947 +0000 UTC m=+1760.464321116" Oct 31 03:24:39.775573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:39.775536 4456 generic.go:356] "Generic (PLEG): container finished" podID="d1497ff1-e950-4e9e-bdd3-d8b80036744d" containerID="265db5e0d8ebbb50c5f7d411ff2f29a15cbc75b8bbbdabadee5282e2ac5dd923" exitCode=0 Oct 31 03:24:39.776028 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:39.775623 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" event={"ID":"d1497ff1-e950-4e9e-bdd3-d8b80036744d","Type":"ContainerDied","Data":"265db5e0d8ebbb50c5f7d411ff2f29a15cbc75b8bbbdabadee5282e2ac5dd923"} Oct 31 03:24:39.933904 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:39.933879 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:40.093429 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.093334 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-xks-ca\") pod \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " Oct 31 03:24:40.093593 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.093471 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-openshift-ca\") pod \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " Oct 31 03:24:40.093671 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.093649 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-kube-api-access-9gbnz\" (UniqueName: \"kubernetes.io/projected/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-kube-api-access-9gbnz\") pod \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\" (UID: \"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2\") " Oct 31 03:24:40.093932 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.093905 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-openshift-ca" pod "4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2" (UID: "4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:40.094257 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.094229 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:40.095586 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.095564 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-xks-ca" pod "4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2" (UID: "4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:40.095668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.095637 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-kube-api-access-9gbnz" pod "4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2" (UID: "4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2"). InnerVolumeSpecName "kube-api-access-9gbnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:40.194955 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.194915 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:40.194955 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.194953 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gbnz\" (UniqueName: \"kubernetes.io/projected/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2-kube-api-access-9gbnz\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:40.787384 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.787351 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" Oct 31 03:24:40.787832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.787382 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt" event={"ID":"4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2","Type":"ContainerDied","Data":"4a8f7a2796aa91320bb93194e1d81dddc3a6db8d8b64a9454b262b1e4556190a"} Oct 31 03:24:40.787832 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.787423 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a8f7a2796aa91320bb93194e1d81dddc3a6db8d8b64a9454b262b1e4556190a" Oct 31 03:24:40.992857 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:40.991860 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:41.041738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.041713 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:24:41.107967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.107931 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-openshift-ca\") pod \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " Oct 31 03:24:41.108129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.107997 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1497ff1-e950-4e9e-bdd3-d8b80036744d-kube-api-access-vsccr\" (UniqueName: \"kubernetes.io/projected/d1497ff1-e950-4e9e-bdd3-d8b80036744d-kube-api-access-vsccr\") pod \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " Oct 31 03:24:41.108129 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.108100 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-xks-ca\") pod \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\" (UID: \"d1497ff1-e950-4e9e-bdd3-d8b80036744d\") " Oct 31 03:24:41.108348 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.108326 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-openshift-ca" pod "d1497ff1-e950-4e9e-bdd3-d8b80036744d" (UID: "d1497ff1-e950-4e9e-bdd3-d8b80036744d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:41.108648 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.108628 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.111434 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.111403 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-xks-ca" pod "d1497ff1-e950-4e9e-bdd3-d8b80036744d" (UID: "d1497ff1-e950-4e9e-bdd3-d8b80036744d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:41.112772 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.112743 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1497ff1-e950-4e9e-bdd3-d8b80036744d-kube-api-access-vsccr" pod "d1497ff1-e950-4e9e-bdd3-d8b80036744d" (UID: "d1497ff1-e950-4e9e-bdd3-d8b80036744d"). InnerVolumeSpecName "kube-api-access-vsccr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.145291 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.145260 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:24:41.148987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.148967 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:24:41.210011 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.209979 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-data\") pod \"5b329d07-76ed-4336-be5d-ac8fc49c0276\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " Oct 31 03:24:41.210213 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210029 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b329d07-76ed-4336-be5d-ac8fc49c0276-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b329d07-76ed-4336-be5d-ac8fc49c0276-istiod-ca-cert\") pod \"5b329d07-76ed-4336-be5d-ac8fc49c0276\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " Oct 31 03:24:41.210213 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210080 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-token\" (UniqueName: \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-token\") pod \"5b329d07-76ed-4336-be5d-ac8fc49c0276\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " Oct 31 03:24:41.210213 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210124 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-credential-socket\") pod \"5b329d07-76ed-4336-be5d-ac8fc49c0276\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " Oct 31 03:24:41.210213 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210158 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-envoy\") pod \"5b329d07-76ed-4336-be5d-ac8fc49c0276\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " Oct 31 03:24:41.210213 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210184 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-certs\") pod \"5b329d07-76ed-4336-be5d-ac8fc49c0276\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " Oct 31 03:24:41.210474 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210216 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-socket\") pod \"5b329d07-76ed-4336-be5d-ac8fc49c0276\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " Oct 31 03:24:41.210474 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210332 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-kube-api-access-nrjhd\" (UniqueName: \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-kube-api-access-nrjhd\") pod \"5b329d07-76ed-4336-be5d-ac8fc49c0276\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " Oct 31 03:24:41.210474 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210329 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-data" pod "5b329d07-76ed-4336-be5d-ac8fc49c0276" (UID: "5b329d07-76ed-4336-be5d-ac8fc49c0276"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.210474 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210371 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-podinfo\") pod \"5b329d07-76ed-4336-be5d-ac8fc49c0276\" (UID: \"5b329d07-76ed-4336-be5d-ac8fc49c0276\") " Oct 31 03:24:41.210674 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210561 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-socket" pod "5b329d07-76ed-4336-be5d-ac8fc49c0276" (UID: "5b329d07-76ed-4336-be5d-ac8fc49c0276"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.210674 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210643 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-credential-socket" pod "5b329d07-76ed-4336-be5d-ac8fc49c0276" (UID: "5b329d07-76ed-4336-be5d-ac8fc49c0276"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.210674 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210649 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b329d07-76ed-4336-be5d-ac8fc49c0276-istiod-ca-cert" pod "5b329d07-76ed-4336-be5d-ac8fc49c0276" (UID: "5b329d07-76ed-4336-be5d-ac8fc49c0276"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:41.210917 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210891 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsccr\" (UniqueName: \"kubernetes.io/projected/d1497ff1-e950-4e9e-bdd3-d8b80036744d-kube-api-access-vsccr\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.211010 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210920 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1497ff1-e950-4e9e-bdd3-d8b80036744d-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.211010 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210938 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.211010 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210952 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b329d07-76ed-4336-be5d-ac8fc49c0276-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.211010 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210967 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.211010 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.210981 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.211285 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.211210 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-certs" pod "5b329d07-76ed-4336-be5d-ac8fc49c0276" (UID: "5b329d07-76ed-4336-be5d-ac8fc49c0276"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.212927 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.212900 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-podinfo" pod "5b329d07-76ed-4336-be5d-ac8fc49c0276" (UID: "5b329d07-76ed-4336-be5d-ac8fc49c0276"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:41.213112 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.213091 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-token" pod "5b329d07-76ed-4336-be5d-ac8fc49c0276" (UID: "5b329d07-76ed-4336-be5d-ac8fc49c0276"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.213276 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.213261 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-kube-api-access-nrjhd" pod "5b329d07-76ed-4336-be5d-ac8fc49c0276" (UID: "5b329d07-76ed-4336-be5d-ac8fc49c0276"). InnerVolumeSpecName "kube-api-access-nrjhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.213927 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.213913 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-envoy" pod "5b329d07-76ed-4336-be5d-ac8fc49c0276" (UID: "5b329d07-76ed-4336-be5d-ac8fc49c0276"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.311450 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311358 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istiod-ca-cert\") pod \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " Oct 31 03:24:41.311450 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311439 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-socket\") pod \"81d96e62-46ab-4df9-95b8-da022d4fce60\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " Oct 31 03:24:41.311686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311545 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-socket\") pod \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " Oct 31 03:24:41.311686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311596 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-envoy\") pod \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " Oct 31 03:24:41.311686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311646 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-podinfo\") pod \"81d96e62-46ab-4df9-95b8-da022d4fce60\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " Oct 31 03:24:41.311686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311653 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-socket" pod "81d96e62-46ab-4df9-95b8-da022d4fce60" (UID: "81d96e62-46ab-4df9-95b8-da022d4fce60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.311837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311690 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-data\") pod \"81d96e62-46ab-4df9-95b8-da022d4fce60\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " Oct 31 03:24:41.311837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311720 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-certs\") pod \"81d96e62-46ab-4df9-95b8-da022d4fce60\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " Oct 31 03:24:41.311837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311750 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-credential-socket\") pod \"81d96e62-46ab-4df9-95b8-da022d4fce60\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " Oct 31 03:24:41.311837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311794 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-credential-socket\") pod \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " Oct 31 03:24:41.311965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311845 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-token\" (UniqueName: \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-token\") pod \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " Oct 31 03:24:41.311965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311878 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-certs\") pod \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " Oct 31 03:24:41.311965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311935 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-data\") pod \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " Oct 31 03:24:41.312107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311971 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d96e62-46ab-4df9-95b8-da022d4fce60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d96e62-46ab-4df9-95b8-da022d4fce60-istiod-ca-cert\") pod \"81d96e62-46ab-4df9-95b8-da022d4fce60\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " Oct 31 03:24:41.312107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312000 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-envoy\") pod \"81d96e62-46ab-4df9-95b8-da022d4fce60\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " Oct 31 03:24:41.312107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312031 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-token\" (UniqueName: \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-token\") pod \"81d96e62-46ab-4df9-95b8-da022d4fce60\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " Oct 31 03:24:41.312107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312082 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-kube-api-access-hw4hj\" (UniqueName: \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-kube-api-access-hw4hj\") pod \"81d96e62-46ab-4df9-95b8-da022d4fce60\" (UID: \"81d96e62-46ab-4df9-95b8-da022d4fce60\") " Oct 31 03:24:41.312278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312148 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-kube-api-access-mmlgx\" (UniqueName: \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-kube-api-access-mmlgx\") pod \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " Oct 31 03:24:41.312278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311794 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istiod-ca-cert" pod "927cee97-88f9-4e3f-bc1f-10ffc85658ac" (UID: "927cee97-88f9-4e3f-bc1f-10ffc85658ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:41.312278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311822 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-socket" pod "927cee97-88f9-4e3f-bc1f-10ffc85658ac" (UID: "927cee97-88f9-4e3f-bc1f-10ffc85658ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.312278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.311969 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-credential-socket" pod "81d96e62-46ab-4df9-95b8-da022d4fce60" (UID: "81d96e62-46ab-4df9-95b8-da022d4fce60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.312278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312080 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-data" pod "81d96e62-46ab-4df9-95b8-da022d4fce60" (UID: "81d96e62-46ab-4df9-95b8-da022d4fce60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.312278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312161 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-certs" pod "81d96e62-46ab-4df9-95b8-da022d4fce60" (UID: "81d96e62-46ab-4df9-95b8-da022d4fce60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.312278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312274 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-podinfo\") pod \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\" (UID: \"927cee97-88f9-4e3f-bc1f-10ffc85658ac\") " Oct 31 03:24:41.312573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312348 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-data" pod "927cee97-88f9-4e3f-bc1f-10ffc85658ac" (UID: "927cee97-88f9-4e3f-bc1f-10ffc85658ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.312573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312419 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-credential-socket" pod "927cee97-88f9-4e3f-bc1f-10ffc85658ac" (UID: "927cee97-88f9-4e3f-bc1f-10ffc85658ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.312794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312767 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d96e62-46ab-4df9-95b8-da022d4fce60-istiod-ca-cert" pod "81d96e62-46ab-4df9-95b8-da022d4fce60" (UID: "81d96e62-46ab-4df9-95b8-da022d4fce60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:41.312926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312815 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.312926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312840 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrjhd\" (UniqueName: \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-kube-api-access-nrjhd\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.312926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312858 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.312926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312872 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.312926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312887 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.312926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312901 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.312926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312915 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.313394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312930 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.313394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312947 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.313394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312963 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.313394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312977 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.313394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.312992 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.313394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.313005 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b329d07-76ed-4336-be5d-ac8fc49c0276-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.313394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.313313 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-certs" pod "927cee97-88f9-4e3f-bc1f-10ffc85658ac" (UID: "927cee97-88f9-4e3f-bc1f-10ffc85658ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.314736 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.314669 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-envoy" pod "927cee97-88f9-4e3f-bc1f-10ffc85658ac" (UID: "927cee97-88f9-4e3f-bc1f-10ffc85658ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.315027 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.314999 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-token" pod "927cee97-88f9-4e3f-bc1f-10ffc85658ac" (UID: "927cee97-88f9-4e3f-bc1f-10ffc85658ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.315184 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.315097 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-token" pod "81d96e62-46ab-4df9-95b8-da022d4fce60" (UID: "81d96e62-46ab-4df9-95b8-da022d4fce60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.315340 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.315316 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-kube-api-access-hw4hj" pod "81d96e62-46ab-4df9-95b8-da022d4fce60" (UID: "81d96e62-46ab-4df9-95b8-da022d4fce60"). InnerVolumeSpecName "kube-api-access-hw4hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.315440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.315423 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-podinfo" pod "927cee97-88f9-4e3f-bc1f-10ffc85658ac" (UID: "927cee97-88f9-4e3f-bc1f-10ffc85658ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:41.315788 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.315753 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-kube-api-access-mmlgx" pod "927cee97-88f9-4e3f-bc1f-10ffc85658ac" (UID: "927cee97-88f9-4e3f-bc1f-10ffc85658ac"). InnerVolumeSpecName "kube-api-access-mmlgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.315904 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.315783 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-podinfo" pod "81d96e62-46ab-4df9-95b8-da022d4fce60" (UID: "81d96e62-46ab-4df9-95b8-da022d4fce60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:41.316351 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.316330 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-envoy" pod "81d96e62-46ab-4df9-95b8-da022d4fce60" (UID: "81d96e62-46ab-4df9-95b8-da022d4fce60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.414099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.414066 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.414099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.414095 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.414099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.414104 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.414334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.414113 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.414334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.414122 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927cee97-88f9-4e3f-bc1f-10ffc85658ac-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.414334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.414130 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d96e62-46ab-4df9-95b8-da022d4fce60-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.414334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.414141 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.414334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.414149 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.414334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.414158 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw4hj\" (UniqueName: \"kubernetes.io/projected/81d96e62-46ab-4df9-95b8-da022d4fce60-kube-api-access-hw4hj\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.414334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.414168 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmlgx\" (UniqueName: \"kubernetes.io/projected/927cee97-88f9-4e3f-bc1f-10ffc85658ac-kube-api-access-mmlgx\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.796966 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.796933 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" Oct 31 03:24:41.796966 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.796961 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8" event={"ID":"d1497ff1-e950-4e9e-bdd3-d8b80036744d","Type":"ContainerDied","Data":"4fe28b6c054cbbb105a22d3e3c33775667703dbc84195804ebe7a532c7705701"} Oct 31 03:24:41.798096 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.797014 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fe28b6c054cbbb105a22d3e3c33775667703dbc84195804ebe7a532c7705701" Oct 31 03:24:41.798731 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.798702 4456 generic.go:356] "Generic (PLEG): container finished" podID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerID="2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda" exitCode=0 Oct 31 03:24:41.798837 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.798769 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" Oct 31 03:24:41.799175 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.799147 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" event={"ID":"927cee97-88f9-4e3f-bc1f-10ffc85658ac","Type":"ContainerDied","Data":"2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda"} Oct 31 03:24:41.799293 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.799208 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp" event={"ID":"927cee97-88f9-4e3f-bc1f-10ffc85658ac","Type":"ContainerDied","Data":"330be89ada2b13dec3a4a610e9e747ea1b31394f56dd54b97162e4d4f473c8a5"} Oct 31 03:24:41.799293 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.799243 4456 scope.go:122] "RemoveContainer" containerID="2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda" Oct 31 03:24:41.803081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.803056 4456 generic.go:356] "Generic (PLEG): container finished" podID="81d96e62-46ab-4df9-95b8-da022d4fce60" containerID="98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746" exitCode=0 Oct 31 03:24:41.803199 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.803120 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" Oct 31 03:24:41.803273 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.803211 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" event={"ID":"81d96e62-46ab-4df9-95b8-da022d4fce60","Type":"ContainerDied","Data":"98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746"} Oct 31 03:24:41.803441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.803424 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9" event={"ID":"81d96e62-46ab-4df9-95b8-da022d4fce60","Type":"ContainerDied","Data":"4c51bab0e8c4ed27139b609efeddce77a661e3268b4b49d71d61e0c3cf0146e9"} Oct 31 03:24:41.804716 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.804696 4456 generic.go:356] "Generic (PLEG): container finished" podID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerID="aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4" exitCode=0 Oct 31 03:24:41.804822 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.804736 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" Oct 31 03:24:41.804822 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.804779 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" event={"ID":"5b329d07-76ed-4336-be5d-ac8fc49c0276","Type":"ContainerDied","Data":"aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4"} Oct 31 03:24:41.804894 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.804828 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx" event={"ID":"5b329d07-76ed-4336-be5d-ac8fc49c0276","Type":"ContainerDied","Data":"5af9d7a1bc0d126280a9aaa21b1e5c448dfb888fe47b0e3ef177eda810ecc2f2"} Oct 31 03:24:41.862404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.846494 4456 scope.go:122] "RemoveContainer" containerID="2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda" Oct 31 03:24:41.862404 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:41.846865 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda\": container with ID starting with 2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda not found: ID does not exist" containerID="2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda" Oct 31 03:24:41.862404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.846896 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda"} err="failed to get container status \"2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda\": rpc error: code = NotFound desc = could not find container \"2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda\": container with ID starting with 2e5f9f5bf01267dfdfd6bc6644eca3d381395a569d9204d5ba69166f54283bda not found: ID does not exist" Oct 31 03:24:41.862404 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.846916 4456 scope.go:122] "RemoveContainer" containerID="98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746" Oct 31 03:24:41.872849 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.872762 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9"] Oct 31 03:24:41.880451 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.880197 4456 scope.go:122] "RemoveContainer" containerID="98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746" Oct 31 03:24:41.881639 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:41.880592 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746\": container with ID starting with 98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746 not found: ID does not exist" containerID="98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746" Oct 31 03:24:41.881639 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.880630 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746"} err="failed to get container status \"98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746\": rpc error: code = NotFound desc = could not find container \"98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746\": container with ID starting with 98edc7de084de1cddfbd180d2d2a2e623bfce00b551b00978686b9b6a9958746 not found: ID does not exist" Oct 31 03:24:41.881639 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.880655 4456 scope.go:122] "RemoveContainer" containerID="aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4" Oct 31 03:24:41.881639 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.881577 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-svpx9"] Oct 31 03:24:41.904323 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.900088 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp"] Oct 31 03:24:41.906926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.906899 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-8xxzp"] Oct 31 03:24:41.913535 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.913446 4456 scope.go:122] "RemoveContainer" containerID="aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4" Oct 31 03:24:41.923359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.921222 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx"] Oct 31 03:24:41.923359 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:41.923093 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4\": container with ID starting with aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4 not found: ID does not exist" containerID="aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4" Oct 31 03:24:41.923359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.923140 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4"} err="failed to get container status \"aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4\": rpc error: code = NotFound desc = could not find container \"aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4\": container with ID starting with aff5df51b1d40a8d06fdc68d425c14b7fb3440e5c1b697d8dac684a5658bc3b4 not found: ID does not exist" Oct 31 03:24:41.927266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:41.927239 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-lglfx"] Oct 31 03:24:42.030810 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.030782 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:24:42.103177 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.103147 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:24:42.122506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122475 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-socket\") pod \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " Oct 31 03:24:42.122506 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122507 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-token\" (UniqueName: \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-token\") pod \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " Oct 31 03:24:42.122748 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122567 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-podinfo\") pod \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " Oct 31 03:24:42.122748 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122600 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-certs\") pod \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " Oct 31 03:24:42.122748 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122654 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-kube-api-access-xgfjk\" (UniqueName: \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-kube-api-access-xgfjk\") pod \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " Oct 31 03:24:42.122748 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122713 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-envoy\") pod \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " Oct 31 03:24:42.122975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122783 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istiod-ca-cert\") pod \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " Oct 31 03:24:42.122975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122807 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-socket" pod "c962528e-e5f8-4c4a-a8d1-5220b596fdf4" (UID: "c962528e-e5f8-4c4a-a8d1-5220b596fdf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.122975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122816 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-credential-socket\") pod \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " Oct 31 03:24:42.122975 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122932 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-data\") pod \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\" (UID: \"c962528e-e5f8-4c4a-a8d1-5220b596fdf4\") " Oct 31 03:24:42.123259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.122982 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-certs" pod "c962528e-e5f8-4c4a-a8d1-5220b596fdf4" (UID: "c962528e-e5f8-4c4a-a8d1-5220b596fdf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.123259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.123008 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-credential-socket" pod "c962528e-e5f8-4c4a-a8d1-5220b596fdf4" (UID: "c962528e-e5f8-4c4a-a8d1-5220b596fdf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.123349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.123257 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-data" pod "c962528e-e5f8-4c4a-a8d1-5220b596fdf4" (UID: "c962528e-e5f8-4c4a-a8d1-5220b596fdf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.123349 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.123286 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istiod-ca-cert" pod "c962528e-e5f8-4c4a-a8d1-5220b596fdf4" (UID: "c962528e-e5f8-4c4a-a8d1-5220b596fdf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:42.123623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.123527 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.123623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.123553 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.123623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.123568 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.123623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.123582 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.123623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.123596 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.124867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.124844 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-podinfo" pod "c962528e-e5f8-4c4a-a8d1-5220b596fdf4" (UID: "c962528e-e5f8-4c4a-a8d1-5220b596fdf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:42.125156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.125137 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-token" pod "c962528e-e5f8-4c4a-a8d1-5220b596fdf4" (UID: "c962528e-e5f8-4c4a-a8d1-5220b596fdf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.125156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.125148 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-kube-api-access-xgfjk" pod "c962528e-e5f8-4c4a-a8d1-5220b596fdf4" (UID: "c962528e-e5f8-4c4a-a8d1-5220b596fdf4"). InnerVolumeSpecName "kube-api-access-xgfjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.125350 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.125332 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-envoy" pod "c962528e-e5f8-4c4a-a8d1-5220b596fdf4" (UID: "c962528e-e5f8-4c4a-a8d1-5220b596fdf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.224791 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.224705 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-socket\") pod \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " Oct 31 03:24:42.224791 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.224761 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-kube-api-access-7fzzp\" (UniqueName: \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-kube-api-access-7fzzp\") pod \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " Oct 31 03:24:42.224967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.224806 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istiod-ca-cert\") pod \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " Oct 31 03:24:42.224967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.224832 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-envoy\") pod \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " Oct 31 03:24:42.224967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.224871 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-certs\") pod \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " Oct 31 03:24:42.224967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.224946 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-credential-socket\") pod \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " Oct 31 03:24:42.225196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.224978 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-data\") pod \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " Oct 31 03:24:42.225196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225061 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-socket" pod "b9ab1680-3c8d-48aa-a226-3d420b0d98ad" (UID: "b9ab1680-3c8d-48aa-a226-3d420b0d98ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.225196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225175 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-token\" (UniqueName: \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-token\") pod \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " Oct 31 03:24:42.225352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225209 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-podinfo\") pod \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\" (UID: \"b9ab1680-3c8d-48aa-a226-3d420b0d98ad\") " Oct 31 03:24:42.225352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225291 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-certs" pod "b9ab1680-3c8d-48aa-a226-3d420b0d98ad" (UID: "b9ab1680-3c8d-48aa-a226-3d420b0d98ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.225352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225305 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-data" pod "b9ab1680-3c8d-48aa-a226-3d420b0d98ad" (UID: "b9ab1680-3c8d-48aa-a226-3d420b0d98ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.225352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225314 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istiod-ca-cert" pod "b9ab1680-3c8d-48aa-a226-3d420b0d98ad" (UID: "b9ab1680-3c8d-48aa-a226-3d420b0d98ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:42.225352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225332 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-credential-socket" pod "b9ab1680-3c8d-48aa-a226-3d420b0d98ad" (UID: "b9ab1680-3c8d-48aa-a226-3d420b0d98ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.225624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225595 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.225624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225610 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.225624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225619 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.225749 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225630 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.225749 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225638 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.225749 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225646 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.225749 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225656 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.225749 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225665 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgfjk\" (UniqueName: \"kubernetes.io/projected/c962528e-e5f8-4c4a-a8d1-5220b596fdf4-kube-api-access-xgfjk\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.225749 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.225677 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.227727 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.227689 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-token" pod "b9ab1680-3c8d-48aa-a226-3d420b0d98ad" (UID: "b9ab1680-3c8d-48aa-a226-3d420b0d98ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.227884 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.227858 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-kube-api-access-7fzzp" pod "b9ab1680-3c8d-48aa-a226-3d420b0d98ad" (UID: "b9ab1680-3c8d-48aa-a226-3d420b0d98ad"). InnerVolumeSpecName "kube-api-access-7fzzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.228438 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.228411 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-envoy" pod "b9ab1680-3c8d-48aa-a226-3d420b0d98ad" (UID: "b9ab1680-3c8d-48aa-a226-3d420b0d98ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.228587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.228568 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-podinfo" pod "b9ab1680-3c8d-48aa-a226-3d420b0d98ad" (UID: "b9ab1680-3c8d-48aa-a226-3d420b0d98ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:42.327266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.327226 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.327266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.327268 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.327266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.327285 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fzzp\" (UniqueName: \"kubernetes.io/projected/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-kube-api-access-7fzzp\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.327619 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.327300 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ab1680-3c8d-48aa-a226-3d420b0d98ad-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.436188 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.436154 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" path="/var/lib/kubelet/pods/5b329d07-76ed-4336-be5d-ac8fc49c0276/volumes" Oct 31 03:24:42.436641 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.436627 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d96e62-46ab-4df9-95b8-da022d4fce60" path="/var/lib/kubelet/pods/81d96e62-46ab-4df9-95b8-da022d4fce60/volumes" Oct 31 03:24:42.437073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.437054 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" path="/var/lib/kubelet/pods/927cee97-88f9-4e3f-bc1f-10ffc85658ac/volumes" Oct 31 03:24:42.815860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.815809 4456 generic.go:356] "Generic (PLEG): container finished" podID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerID="4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8" exitCode=0 Oct 31 03:24:42.816364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.815878 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" Oct 31 03:24:42.816364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.815888 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" event={"ID":"b9ab1680-3c8d-48aa-a226-3d420b0d98ad","Type":"ContainerDied","Data":"4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8"} Oct 31 03:24:42.816364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.815990 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q" event={"ID":"b9ab1680-3c8d-48aa-a226-3d420b0d98ad","Type":"ContainerDied","Data":"4d17a8ae0f52941524bb6dcf53fcbab66982af6882f4914fe3867309bfd8c40a"} Oct 31 03:24:42.816364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.816017 4456 scope.go:122] "RemoveContainer" containerID="4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8" Oct 31 03:24:42.820320 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.820291 4456 generic.go:356] "Generic (PLEG): container finished" podID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerID="57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a" exitCode=0 Oct 31 03:24:42.820454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.820358 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" Oct 31 03:24:42.820454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.820377 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" event={"ID":"c962528e-e5f8-4c4a-a8d1-5220b596fdf4","Type":"ContainerDied","Data":"57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a"} Oct 31 03:24:42.820454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.820418 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld" event={"ID":"c962528e-e5f8-4c4a-a8d1-5220b596fdf4","Type":"ContainerDied","Data":"6041905b1c832ad109f5caf7f8a7e866d23d197c2f291b6add8d192b9300e28d"} Oct 31 03:24:42.829028 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.829010 4456 scope.go:122] "RemoveContainer" containerID="4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8" Oct 31 03:24:42.829317 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:42.829299 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8\": container with ID starting with 4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8 not found: ID does not exist" containerID="4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8" Oct 31 03:24:42.829366 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.829326 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8"} err="failed to get container status \"4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8\": rpc error: code = NotFound desc = could not find container \"4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8\": container with ID starting with 4a74946941bc6066809cb20f186fa43057d83f4913641254f65bddab80f92ec8 not found: ID does not exist" Oct 31 03:24:42.829366 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.829342 4456 scope.go:122] "RemoveContainer" containerID="57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a" Oct 31 03:24:42.839889 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.839872 4456 scope.go:122] "RemoveContainer" containerID="57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a" Oct 31 03:24:42.840213 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:42.840189 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a\": container with ID starting with 57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a not found: ID does not exist" containerID="57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a" Oct 31 03:24:42.840297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.840224 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a"} err="failed to get container status \"57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a\": rpc error: code = NotFound desc = could not find container \"57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a\": container with ID starting with 57c7e241321c866ba36cc93a940a5d35ebd43aa0d826737fe2f1617400eb796a not found: ID does not exist" Oct 31 03:24:42.842073 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.842047 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q"] Oct 31 03:24:42.845865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.845842 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-rj54q"] Oct 31 03:24:42.864410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.864388 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld"] Oct 31 03:24:42.870238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:42.870217 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-nfxld"] Oct 31 03:24:44.435028 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:44.434995 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" path="/var/lib/kubelet/pods/b9ab1680-3c8d-48aa-a226-3d420b0d98ad/volumes" Oct 31 03:24:44.435560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:44.435543 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" path="/var/lib/kubelet/pods/c962528e-e5f8-4c4a-a8d1-5220b596fdf4/volumes" Oct 31 03:24:46.341619 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:46.341580 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk"] Oct 31 03:24:46.349677 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:46.349648 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8"] Oct 31 03:24:47.787051 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:47.787004 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt"] Oct 31 03:24:47.791230 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:47.791182 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b79c13-4lpzt"] Oct 31 03:24:48.436224 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:48.436183 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2" path="/var/lib/kubelet/pods/4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2/volumes" Oct 31 03:24:49.034870 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:49.034838 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr"] Oct 31 03:24:49.038922 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:49.038894 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd"] Oct 31 03:24:49.042030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:49.042009 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae54720-t9znr"] Oct 31 03:24:49.045020 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:49.044997 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02410db4-v9cvd"] Oct 31 03:24:50.272004 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.271967 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7"] Oct 31 03:24:50.272753 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272730 4456 cpu_manager.go:414] "containerMap: removing container" podUID="d1497ff1-e950-4e9e-bdd3-d8b80036744d" containerName="revoke-keys" Oct 31 03:24:50.272891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272770 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1497ff1-e950-4e9e-bdd3-d8b80036744d" containerName="revoke-keys" Oct 31 03:24:50.272891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272805 4456 cpu_manager.go:414] "containerMap: removing container" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerName="istio-proxy" Oct 31 03:24:50.272891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272813 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerName="istio-proxy" Oct 31 03:24:50.272891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272826 4456 cpu_manager.go:414] "containerMap: removing container" podUID="4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2" containerName="revoke-keys" Oct 31 03:24:50.272891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272835 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2" containerName="revoke-keys" Oct 31 03:24:50.272891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272859 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerName="istio-proxy" Oct 31 03:24:50.272891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272866 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerName="istio-proxy" Oct 31 03:24:50.272891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272877 4456 cpu_manager.go:414] "containerMap: removing container" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerName="istio-proxy" Oct 31 03:24:50.272891 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272883 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerName="istio-proxy" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272923 4456 cpu_manager.go:414] "containerMap: removing container" podUID="81d96e62-46ab-4df9-95b8-da022d4fce60" containerName="istio-proxy" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.272933 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d96e62-46ab-4df9-95b8-da022d4fce60" containerName="istio-proxy" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.273007 4456 cpu_manager.go:414] "containerMap: removing container" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerName="istio-proxy" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.273017 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerName="istio-proxy" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.273181 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1497ff1-e950-4e9e-bdd3-d8b80036744d" containerName="revoke-keys" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.273200 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c962528e-e5f8-4c4a-a8d1-5220b596fdf4" containerName="istio-proxy" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.273214 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3dd074-0a3d-4d5f-8e05-bf0673cd8ee2" containerName="revoke-keys" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.273227 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ab1680-3c8d-48aa-a226-3d420b0d98ad" containerName="istio-proxy" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.273237 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b329d07-76ed-4336-be5d-ac8fc49c0276" containerName="istio-proxy" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.273248 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d96e62-46ab-4df9-95b8-da022d4fce60" containerName="istio-proxy" Oct 31 03:24:50.273374 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.273258 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="927cee97-88f9-4e3f-bc1f-10ffc85658ac" containerName="istio-proxy" Oct 31 03:24:50.279219 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.279189 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.281486 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.280955 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599"] Oct 31 03:24:50.287069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.285808 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.287337 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.287242 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7"] Oct 31 03:24:50.293013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.292993 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599"] Oct 31 03:24:50.293254 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.293237 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:24:50.294008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.293991 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:24:50.426311 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.426273 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a88c35e-26mw7\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.426540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.426343 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-612b4cad-pz599\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.426540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.426453 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a88c35e-26mw7\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.426540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.426502 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74zm\" (UniqueName: \"kubernetes.io/projected/0cf8f72a-f23d-47e9-8097-0b50a6217f91-kube-api-access-s74zm\") pod \"maas-api-revoke-sub-612b4cad-pz599\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.426540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.426527 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-612b4cad-pz599\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.426721 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.426622 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbhrp\" (UniqueName: \"kubernetes.io/projected/ef8a2f48-1a8d-4701-a802-8728187a54dc-kube-api-access-rbhrp\") pod \"maas-api-revoke-sub-7a88c35e-26mw7\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.435828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.435806 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84" path="/var/lib/kubelet/pods/33c98d6d-570d-42ec-ae2c-8b2f7f1c3d84/volumes" Oct 31 03:24:50.436298 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.436283 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54db39e-4c28-4c13-8272-63bdf94286e0" path="/var/lib/kubelet/pods/d54db39e-4c28-4c13-8272-63bdf94286e0/volumes" Oct 31 03:24:50.527764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.527679 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a88c35e-26mw7\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.527764 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.527757 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74zm\" (UniqueName: \"kubernetes.io/projected/0cf8f72a-f23d-47e9-8097-0b50a6217f91-kube-api-access-s74zm\") pod \"maas-api-revoke-sub-612b4cad-pz599\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.528069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.527788 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-612b4cad-pz599\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.528069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.527842 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbhrp\" (UniqueName: \"kubernetes.io/projected/ef8a2f48-1a8d-4701-a802-8728187a54dc-kube-api-access-rbhrp\") pod \"maas-api-revoke-sub-7a88c35e-26mw7\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.528069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.527903 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a88c35e-26mw7\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.528069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.527945 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-612b4cad-pz599\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.528436 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.528408 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a88c35e-26mw7\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.528878 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.528853 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-612b4cad-pz599\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.530445 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.530426 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a88c35e-26mw7\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.530526 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.530441 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-612b4cad-pz599\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.541392 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.541369 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74zm\" (UniqueName: \"kubernetes.io/projected/0cf8f72a-f23d-47e9-8097-0b50a6217f91-kube-api-access-s74zm\") pod \"maas-api-revoke-sub-612b4cad-pz599\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.541516 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.541498 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbhrp\" (UniqueName: \"kubernetes.io/projected/ef8a2f48-1a8d-4701-a802-8728187a54dc-kube-api-access-rbhrp\") pod \"maas-api-revoke-sub-7a88c35e-26mw7\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.611146 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.611111 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:50.620662 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.620642 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:50.932373 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.932338 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599"] Oct 31 03:24:50.934847 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:24:50.934767 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8a2f48_1a8d_4701_a802_8728187a54dc.slice/crio-81010fea3fab2728c84ec6f574c045f5b7e70218790366769dcc5274561f8f2a WatchSource:0}: Error finding container 81010fea3fab2728c84ec6f574c045f5b7e70218790366769dcc5274561f8f2a: Status 404 returned error can't find the container with id 81010fea3fab2728c84ec6f574c045f5b7e70218790366769dcc5274561f8f2a Oct 31 03:24:50.935136 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:24:50.935102 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf8f72a_f23d_47e9_8097_0b50a6217f91.slice/crio-d489b6317b2bbe94aa9a2d170020399ff0d4e19f7093fb1fa5f5a301818fe6cc WatchSource:0}: Error finding container d489b6317b2bbe94aa9a2d170020399ff0d4e19f7093fb1fa5f5a301818fe6cc: Status 404 returned error can't find the container with id d489b6317b2bbe94aa9a2d170020399ff0d4e19f7093fb1fa5f5a301818fe6cc Oct 31 03:24:50.938414 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:50.938394 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7"] Oct 31 03:24:51.343414 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.343355 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" podUID="8f8bee44-c20a-426c-961e-1b369f5046cb" containerName="payload-processing" containerID="cri-o://ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575" gracePeriod=25 Oct 31 03:24:51.350481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.350447 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" podUID="9fd297f3-62d3-4fc8-81ad-7973c85a402e" containerName="payload-pre-processing" containerID="cri-o://dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0" gracePeriod=25 Oct 31 03:24:51.628860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.628830 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:24:51.644969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.644938 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8bee44-c20a-426c-961e-1b369f5046cb-kube-api-access-bkdt9\" (UniqueName: \"kubernetes.io/projected/8f8bee44-c20a-426c-961e-1b369f5046cb-kube-api-access-bkdt9\") pod \"8f8bee44-c20a-426c-961e-1b369f5046cb\" (UID: \"8f8bee44-c20a-426c-961e-1b369f5046cb\") " Oct 31 03:24:51.645170 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.645153 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8bee44-c20a-426c-961e-1b369f5046cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8bee44-c20a-426c-961e-1b369f5046cb-plugins-config-volume\") pod \"8f8bee44-c20a-426c-961e-1b369f5046cb\" (UID: \"8f8bee44-c20a-426c-961e-1b369f5046cb\") " Oct 31 03:24:51.645521 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.645495 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8bee44-c20a-426c-961e-1b369f5046cb-plugins-config-volume" pod "8f8bee44-c20a-426c-961e-1b369f5046cb" (UID: "8f8bee44-c20a-426c-961e-1b369f5046cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:51.648472 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.647285 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8bee44-c20a-426c-961e-1b369f5046cb-kube-api-access-bkdt9" pod "8f8bee44-c20a-426c-961e-1b369f5046cb" (UID: "8f8bee44-c20a-426c-961e-1b369f5046cb"). InnerVolumeSpecName "kube-api-access-bkdt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:51.747070 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.746985 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8bee44-c20a-426c-961e-1b369f5046cb-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:51.747340 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.747315 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkdt9\" (UniqueName: \"kubernetes.io/projected/8f8bee44-c20a-426c-961e-1b369f5046cb-kube-api-access-bkdt9\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:51.951985 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.951852 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" event={"ID":"0cf8f72a-f23d-47e9-8097-0b50a6217f91","Type":"ContainerStarted","Data":"11407a46280cac8a683d1cf1df2160396aeaf64f8fbbb57de63e2ebeb8466155"} Oct 31 03:24:51.951985 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.951968 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" event={"ID":"0cf8f72a-f23d-47e9-8097-0b50a6217f91","Type":"ContainerStarted","Data":"d489b6317b2bbe94aa9a2d170020399ff0d4e19f7093fb1fa5f5a301818fe6cc"} Oct 31 03:24:51.953971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.953938 4456 generic.go:356] "Generic (PLEG): container finished" podID="8f8bee44-c20a-426c-961e-1b369f5046cb" containerID="ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575" exitCode=0 Oct 31 03:24:51.954172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.953995 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" Oct 31 03:24:51.954771 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.954014 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" event={"ID":"8f8bee44-c20a-426c-961e-1b369f5046cb","Type":"ContainerDied","Data":"ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575"} Oct 31 03:24:51.954771 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.954516 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk" event={"ID":"8f8bee44-c20a-426c-961e-1b369f5046cb","Type":"ContainerDied","Data":"db9fddd7a60e89d5d92a7db5dce2f714554c60be71d60378f680496f3419d3da"} Oct 31 03:24:51.954771 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.954550 4456 scope.go:122] "RemoveContainer" containerID="ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575" Oct 31 03:24:51.956813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.956788 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" event={"ID":"ef8a2f48-1a8d-4701-a802-8728187a54dc","Type":"ContainerStarted","Data":"40bfdb356689cc696670d49d4ce9abd19eabe5409c55221d584d4fa9f0941e20"} Oct 31 03:24:51.956913 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.956835 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" event={"ID":"ef8a2f48-1a8d-4701-a802-8728187a54dc","Type":"ContainerStarted","Data":"81010fea3fab2728c84ec6f574c045f5b7e70218790366769dcc5274561f8f2a"} Oct 31 03:24:51.975875 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.975854 4456 scope.go:122] "RemoveContainer" containerID="ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575" Oct 31 03:24:51.976234 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:24:51.976211 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575\": container with ID starting with ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575 not found: ID does not exist" containerID="ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575" Oct 31 03:24:51.976336 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.976240 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575"} err="failed to get container status \"ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575\": rpc error: code = NotFound desc = could not find container \"ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575\": container with ID starting with ecf726462367dff943d263a7f9a50c575b7412fb279e321cfd0f477546fed575 not found: ID does not exist" Oct 31 03:24:51.976714 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.976674 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" podStartSLOduration=1.976661443 podStartE2EDuration="1.976661443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:51.971869277 +0000 UTC m=+1773.652395405" watchObservedRunningTime="2026-10-31 03:24:51.976661443 +0000 UTC m=+1773.657187587" Oct 31 03:24:51.997148 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:51.997025 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" podStartSLOduration=1.997006872 podStartE2EDuration="1.997006872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:51.995517302 +0000 UTC m=+1773.676043429" watchObservedRunningTime="2026-10-31 03:24:51.997006872 +0000 UTC m=+1773.677533017" Oct 31 03:24:52.037514 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:52.037482 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk"] Oct 31 03:24:52.043609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:52.043576 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b79c13-647fb6dd9c-nlwsk"] Oct 31 03:24:52.435794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:52.435760 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8bee44-c20a-426c-961e-1b369f5046cb" path="/var/lib/kubelet/pods/8f8bee44-c20a-426c-961e-1b369f5046cb/volumes" Oct 31 03:24:52.968667 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:52.968625 4456 generic.go:356] "Generic (PLEG): container finished" podID="ef8a2f48-1a8d-4701-a802-8728187a54dc" containerID="40bfdb356689cc696670d49d4ce9abd19eabe5409c55221d584d4fa9f0941e20" exitCode=0 Oct 31 03:24:52.968878 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:52.968702 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" event={"ID":"ef8a2f48-1a8d-4701-a802-8728187a54dc","Type":"ContainerDied","Data":"40bfdb356689cc696670d49d4ce9abd19eabe5409c55221d584d4fa9f0941e20"} Oct 31 03:24:52.970800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:52.970774 4456 generic.go:356] "Generic (PLEG): container finished" podID="0cf8f72a-f23d-47e9-8097-0b50a6217f91" containerID="11407a46280cac8a683d1cf1df2160396aeaf64f8fbbb57de63e2ebeb8466155" exitCode=0 Oct 31 03:24:52.970929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:52.970854 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" event={"ID":"0cf8f72a-f23d-47e9-8097-0b50a6217f91","Type":"ContainerDied","Data":"11407a46280cac8a683d1cf1df2160396aeaf64f8fbbb57de63e2ebeb8466155"} Oct 31 03:24:54.159858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.159834 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:54.174118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.174092 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-openshift-ca\") pod \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " Oct 31 03:24:54.174292 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.174279 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-xks-ca\") pod \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " Oct 31 03:24:54.174343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.174324 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf8f72a-f23d-47e9-8097-0b50a6217f91-kube-api-access-s74zm\" (UniqueName: \"kubernetes.io/projected/0cf8f72a-f23d-47e9-8097-0b50a6217f91-kube-api-access-s74zm\") pod \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\" (UID: \"0cf8f72a-f23d-47e9-8097-0b50a6217f91\") " Oct 31 03:24:54.174550 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.174476 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-openshift-ca" pod "0cf8f72a-f23d-47e9-8097-0b50a6217f91" (UID: "0cf8f72a-f23d-47e9-8097-0b50a6217f91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:54.174765 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.174677 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:54.176660 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.176638 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf8f72a-f23d-47e9-8097-0b50a6217f91-kube-api-access-s74zm" pod "0cf8f72a-f23d-47e9-8097-0b50a6217f91" (UID: "0cf8f72a-f23d-47e9-8097-0b50a6217f91"). InnerVolumeSpecName "kube-api-access-s74zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:54.176856 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.176837 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-xks-ca" pod "0cf8f72a-f23d-47e9-8097-0b50a6217f91" (UID: "0cf8f72a-f23d-47e9-8097-0b50a6217f91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:54.201788 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.201757 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:54.275131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.275011 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-openshift-ca\") pod \"ef8a2f48-1a8d-4701-a802-8728187a54dc\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " Oct 31 03:24:54.275131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.275100 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8a2f48-1a8d-4701-a802-8728187a54dc-kube-api-access-rbhrp\" (UniqueName: \"kubernetes.io/projected/ef8a2f48-1a8d-4701-a802-8728187a54dc-kube-api-access-rbhrp\") pod \"ef8a2f48-1a8d-4701-a802-8728187a54dc\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " Oct 31 03:24:54.275397 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.275284 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-xks-ca\") pod \"ef8a2f48-1a8d-4701-a802-8728187a54dc\" (UID: \"ef8a2f48-1a8d-4701-a802-8728187a54dc\") " Oct 31 03:24:54.275458 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.275394 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-openshift-ca" pod "ef8a2f48-1a8d-4701-a802-8728187a54dc" (UID: "ef8a2f48-1a8d-4701-a802-8728187a54dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:54.275685 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.275663 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf8f72a-f23d-47e9-8097-0b50a6217f91-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:54.275685 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.275686 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s74zm\" (UniqueName: \"kubernetes.io/projected/0cf8f72a-f23d-47e9-8097-0b50a6217f91-kube-api-access-s74zm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:54.275882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.275701 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:54.277953 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.277927 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-xks-ca" pod "ef8a2f48-1a8d-4701-a802-8728187a54dc" (UID: "ef8a2f48-1a8d-4701-a802-8728187a54dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:54.278082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.277968 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8a2f48-1a8d-4701-a802-8728187a54dc-kube-api-access-rbhrp" pod "ef8a2f48-1a8d-4701-a802-8728187a54dc" (UID: "ef8a2f48-1a8d-4701-a802-8728187a54dc"). InnerVolumeSpecName "kube-api-access-rbhrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:54.376657 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.376611 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8a2f48-1a8d-4701-a802-8728187a54dc-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:54.376657 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.376649 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbhrp\" (UniqueName: \"kubernetes.io/projected/ef8a2f48-1a8d-4701-a802-8728187a54dc-kube-api-access-rbhrp\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:55.000058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.999579 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" event={"ID":"0cf8f72a-f23d-47e9-8097-0b50a6217f91","Type":"ContainerDied","Data":"d489b6317b2bbe94aa9a2d170020399ff0d4e19f7093fb1fa5f5a301818fe6cc"} Oct 31 03:24:55.000058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.999690 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d489b6317b2bbe94aa9a2d170020399ff0d4e19f7093fb1fa5f5a301818fe6cc" Oct 31 03:24:55.000058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:54.999799 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599" Oct 31 03:24:55.007059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:55.001931 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" event={"ID":"ef8a2f48-1a8d-4701-a802-8728187a54dc","Type":"ContainerDied","Data":"81010fea3fab2728c84ec6f574c045f5b7e70218790366769dcc5274561f8f2a"} Oct 31 03:24:55.007059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:55.002001 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81010fea3fab2728c84ec6f574c045f5b7e70218790366769dcc5274561f8f2a" Oct 31 03:24:55.007059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:55.004459 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7" Oct 31 03:24:56.812395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.812356 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6"] Oct 31 03:24:56.812892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.812878 4456 cpu_manager.go:414] "containerMap: removing container" podUID="ef8a2f48-1a8d-4701-a802-8728187a54dc" containerName="revoke-keys" Oct 31 03:24:56.812938 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.812893 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8a2f48-1a8d-4701-a802-8728187a54dc" containerName="revoke-keys" Oct 31 03:24:56.812938 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.812904 4456 cpu_manager.go:414] "containerMap: removing container" podUID="8f8bee44-c20a-426c-961e-1b369f5046cb" containerName="payload-processing" Oct 31 03:24:56.812938 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.812909 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8bee44-c20a-426c-961e-1b369f5046cb" containerName="payload-processing" Oct 31 03:24:56.812938 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.812929 4456 cpu_manager.go:414] "containerMap: removing container" podUID="0cf8f72a-f23d-47e9-8097-0b50a6217f91" containerName="revoke-keys" Oct 31 03:24:56.812938 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.812935 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf8f72a-f23d-47e9-8097-0b50a6217f91" containerName="revoke-keys" Oct 31 03:24:56.813113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.813003 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf8f72a-f23d-47e9-8097-0b50a6217f91" containerName="revoke-keys" Oct 31 03:24:56.813113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.813018 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8a2f48-1a8d-4701-a802-8728187a54dc" containerName="revoke-keys" Oct 31 03:24:56.813113 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.813025 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8bee44-c20a-426c-961e-1b369f5046cb" containerName="payload-processing" Oct 31 03:24:56.816644 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.816623 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:56.827814 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.827788 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6"] Oct 31 03:24:56.903984 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.903952 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzx4z\" (UniqueName: \"kubernetes.io/projected/4438eea7-a68f-4d92-9328-9c45998fb8b0-kube-api-access-vzx4z\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:56.904194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.904022 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:56.904194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.904177 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:56.916758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.916731 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:24:56.916758 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:56.916736 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:24:57.005322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:57.005279 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzx4z\" (UniqueName: \"kubernetes.io/projected/4438eea7-a68f-4d92-9328-9c45998fb8b0-kube-api-access-vzx4z\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:57.005531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:57.005353 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:57.005531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:57.005428 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:57.006305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:57.006272 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:57.009202 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:57.009122 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:57.022909 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:57.020333 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzx4z\" (UniqueName: \"kubernetes.io/projected/4438eea7-a68f-4d92-9328-9c45998fb8b0-kube-api-access-vzx4z\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:57.133581 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:57.133496 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:24:57.424953 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:24:57.424912 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4438eea7_a68f_4d92_9328_9c45998fb8b0.slice/crio-5cc8642f3cebc880a284c741ef6e5634465f9e384833cd29f7517a3f49fe45f1 WatchSource:0}: Error finding container 5cc8642f3cebc880a284c741ef6e5634465f9e384833cd29f7517a3f49fe45f1: Status 404 returned error can't find the container with id 5cc8642f3cebc880a284c741ef6e5634465f9e384833cd29f7517a3f49fe45f1 Oct 31 03:24:57.425152 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:57.425126 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6"] Oct 31 03:24:57.526379 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:57.526339 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" podUID="9fd297f3-62d3-4fc8-81ad-7973c85a402e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:58.044473 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:58.044365 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" event={"ID":"4438eea7-a68f-4d92-9328-9c45998fb8b0","Type":"ContainerStarted","Data":"ea393aaa7584af4609381f65828ca15abad48684cc63193da35c504a38e66c00"} Oct 31 03:24:58.044473 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:58.044432 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" event={"ID":"4438eea7-a68f-4d92-9328-9c45998fb8b0","Type":"ContainerStarted","Data":"5cc8642f3cebc880a284c741ef6e5634465f9e384833cd29f7517a3f49fe45f1"} Oct 31 03:24:58.068358 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:58.068297 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" podStartSLOduration=2.068283023 podStartE2EDuration="2.068283023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:58.064461813 +0000 UTC m=+1779.744987952" watchObservedRunningTime="2026-10-31 03:24:58.068283023 +0000 UTC m=+1779.748809150" Oct 31 03:24:59.056278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:59.056242 4456 generic.go:356] "Generic (PLEG): container finished" podID="4438eea7-a68f-4d92-9328-9c45998fb8b0" containerID="ea393aaa7584af4609381f65828ca15abad48684cc63193da35c504a38e66c00" exitCode=0 Oct 31 03:24:59.056736 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:24:59.056405 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" event={"ID":"4438eea7-a68f-4d92-9328-9c45998fb8b0","Type":"ContainerDied","Data":"ea393aaa7584af4609381f65828ca15abad48684cc63193da35c504a38e66c00"} Oct 31 03:25:00.234092 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:00.234060 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:25:00.344736 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:00.344659 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-openshift-ca\") pod \"4438eea7-a68f-4d92-9328-9c45998fb8b0\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " Oct 31 03:25:00.344736 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:00.344716 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4438eea7-a68f-4d92-9328-9c45998fb8b0-kube-api-access-vzx4z\" (UniqueName: \"kubernetes.io/projected/4438eea7-a68f-4d92-9328-9c45998fb8b0-kube-api-access-vzx4z\") pod \"4438eea7-a68f-4d92-9328-9c45998fb8b0\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " Oct 31 03:25:00.344944 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:00.344742 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-xks-ca\") pod \"4438eea7-a68f-4d92-9328-9c45998fb8b0\" (UID: \"4438eea7-a68f-4d92-9328-9c45998fb8b0\") " Oct 31 03:25:00.345061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:00.345020 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-openshift-ca" pod "4438eea7-a68f-4d92-9328-9c45998fb8b0" (UID: "4438eea7-a68f-4d92-9328-9c45998fb8b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:00.345270 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:00.345239 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:00.346835 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:00.346803 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-xks-ca" pod "4438eea7-a68f-4d92-9328-9c45998fb8b0" (UID: "4438eea7-a68f-4d92-9328-9c45998fb8b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:00.347126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:00.347099 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4438eea7-a68f-4d92-9328-9c45998fb8b0-kube-api-access-vzx4z" pod "4438eea7-a68f-4d92-9328-9c45998fb8b0" (UID: "4438eea7-a68f-4d92-9328-9c45998fb8b0"). InnerVolumeSpecName "kube-api-access-vzx4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:00.446407 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:00.446378 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzx4z\" (UniqueName: \"kubernetes.io/projected/4438eea7-a68f-4d92-9328-9c45998fb8b0-kube-api-access-vzx4z\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:00.446407 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:00.446406 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4438eea7-a68f-4d92-9328-9c45998fb8b0-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.076734 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:01.076701 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" Oct 31 03:25:01.076734 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:01.076719 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6" event={"ID":"4438eea7-a68f-4d92-9328-9c45998fb8b0","Type":"ContainerDied","Data":"5cc8642f3cebc880a284c741ef6e5634465f9e384833cd29f7517a3f49fe45f1"} Oct 31 03:25:01.076951 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:01.076775 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cc8642f3cebc880a284c741ef6e5634465f9e384833cd29f7517a3f49fe45f1" Oct 31 03:25:02.135078 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:25:02.135023 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:25:06.934742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:06.934712 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8"] Oct 31 03:25:06.941448 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:06.941417 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-924ee5-pd6v8"] Oct 31 03:25:07.039589 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:07.039552 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b"] Oct 31 03:25:07.526263 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:07.526222 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" podUID="9fd297f3-62d3-4fc8-81ad-7973c85a402e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:25:08.435515 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:08.435483 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1497ff1-e950-4e9e-bdd3-d8b80036744d" path="/var/lib/kubelet/pods/d1497ff1-e950-4e9e-bdd3-d8b80036744d/volumes" Oct 31 03:25:10.412000 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:10.411965 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6"] Oct 31 03:25:10.417936 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:10.417897 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2206e7-drms6"] Oct 31 03:25:10.442199 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:10.442165 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4438eea7-a68f-4d92-9328-9c45998fb8b0" path="/var/lib/kubelet/pods/4438eea7-a68f-4d92-9328-9c45998fb8b0/volumes" Oct 31 03:25:11.620133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:11.620065 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:25:11.786004 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:11.785969 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd297f3-62d3-4fc8-81ad-7973c85a402e-kube-api-access-4czbs\" (UniqueName: \"kubernetes.io/projected/9fd297f3-62d3-4fc8-81ad-7973c85a402e-kube-api-access-4czbs\") pod \"9fd297f3-62d3-4fc8-81ad-7973c85a402e\" (UID: \"9fd297f3-62d3-4fc8-81ad-7973c85a402e\") " Oct 31 03:25:11.786206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:11.786134 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd297f3-62d3-4fc8-81ad-7973c85a402e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd297f3-62d3-4fc8-81ad-7973c85a402e-plugins-config-volume\") pod \"9fd297f3-62d3-4fc8-81ad-7973c85a402e\" (UID: \"9fd297f3-62d3-4fc8-81ad-7973c85a402e\") " Oct 31 03:25:11.786533 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:11.786508 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd297f3-62d3-4fc8-81ad-7973c85a402e-plugins-config-volume" pod "9fd297f3-62d3-4fc8-81ad-7973c85a402e" (UID: "9fd297f3-62d3-4fc8-81ad-7973c85a402e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:11.788606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:11.788579 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd297f3-62d3-4fc8-81ad-7973c85a402e-kube-api-access-4czbs" pod "9fd297f3-62d3-4fc8-81ad-7973c85a402e" (UID: "9fd297f3-62d3-4fc8-81ad-7973c85a402e"). InnerVolumeSpecName "kube-api-access-4czbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:11.887459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:11.887370 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd297f3-62d3-4fc8-81ad-7973c85a402e-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:11.887459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:11.887410 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czbs\" (UniqueName: \"kubernetes.io/projected/9fd297f3-62d3-4fc8-81ad-7973c85a402e-kube-api-access-4czbs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:12.040483 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.040429 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" podUID="6986cbad-3cdf-4333-b1ab-d49bc8bc3eea" containerName="payload-pre-processing" containerID="cri-o://c6ac8d53a441ea9dcdaf99a529d5fe577a01748dcee0a5dd267a5ca105a4f37f" gracePeriod=25 Oct 31 03:25:12.179895 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.179859 4456 generic.go:356] "Generic (PLEG): container finished" podID="6986cbad-3cdf-4333-b1ab-d49bc8bc3eea" containerID="c6ac8d53a441ea9dcdaf99a529d5fe577a01748dcee0a5dd267a5ca105a4f37f" exitCode=0 Oct 31 03:25:12.180090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.179946 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" event={"ID":"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea","Type":"ContainerDied","Data":"c6ac8d53a441ea9dcdaf99a529d5fe577a01748dcee0a5dd267a5ca105a4f37f"} Oct 31 03:25:12.181715 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.181684 4456 generic.go:356] "Generic (PLEG): container finished" podID="9fd297f3-62d3-4fc8-81ad-7973c85a402e" containerID="dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0" exitCode=0 Oct 31 03:25:12.181865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.181752 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" Oct 31 03:25:12.181927 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.181892 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" event={"ID":"9fd297f3-62d3-4fc8-81ad-7973c85a402e","Type":"ContainerDied","Data":"dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0"} Oct 31 03:25:12.181981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.181936 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8" event={"ID":"9fd297f3-62d3-4fc8-81ad-7973c85a402e","Type":"ContainerDied","Data":"875aac3072b8ecc2f466d63a407b967078459166ce6a2d876c6492cbd916e9e0"} Oct 31 03:25:12.181981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.181965 4456 scope.go:122] "RemoveContainer" containerID="dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0" Oct 31 03:25:12.198720 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.198699 4456 scope.go:122] "RemoveContainer" containerID="dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0" Oct 31 03:25:12.199016 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:25:12.198989 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0\": container with ID starting with dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0 not found: ID does not exist" containerID="dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0" Oct 31 03:25:12.199114 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.199029 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0"} err="failed to get container status \"dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0\": rpc error: code = NotFound desc = could not find container \"dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0\": container with ID starting with dd5498638822ddbdfb5dd523e45c0d27085f68c8af4946428d62fd4bab3a72f0 not found: ID does not exist" Oct 31 03:25:12.215206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.215179 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8"] Oct 31 03:25:12.219311 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.219285 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b79c13-56bf85879-6fqh8"] Oct 31 03:25:12.302174 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.302150 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:25:12.392429 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.392396 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-kube-api-access-mdvgm\" (UniqueName: \"kubernetes.io/projected/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-kube-api-access-mdvgm\") pod \"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea\" (UID: \"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea\") " Oct 31 03:25:12.392607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.392477 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-plugins-config-volume\") pod \"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea\" (UID: \"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea\") " Oct 31 03:25:12.392921 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.392886 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-plugins-config-volume" pod "6986cbad-3cdf-4333-b1ab-d49bc8bc3eea" (UID: "6986cbad-3cdf-4333-b1ab-d49bc8bc3eea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:12.393079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.393010 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:12.394738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.394706 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-kube-api-access-mdvgm" pod "6986cbad-3cdf-4333-b1ab-d49bc8bc3eea" (UID: "6986cbad-3cdf-4333-b1ab-d49bc8bc3eea"). InnerVolumeSpecName "kube-api-access-mdvgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:12.436489 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.436416 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd297f3-62d3-4fc8-81ad-7973c85a402e" path="/var/lib/kubelet/pods/9fd297f3-62d3-4fc8-81ad-7973c85a402e/volumes" Oct 31 03:25:12.494014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:12.493982 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdvgm\" (UniqueName: \"kubernetes.io/projected/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea-kube-api-access-mdvgm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:13.191893 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:13.191867 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" Oct 31 03:25:13.192411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:13.191864 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b" event={"ID":"6986cbad-3cdf-4333-b1ab-d49bc8bc3eea","Type":"ContainerDied","Data":"61666d8b775a6b15acb26d0ff9e2c500d5ef777c71cc1aff4e734e85fd6e267f"} Oct 31 03:25:13.192411 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:13.192010 4456 scope.go:122] "RemoveContainer" containerID="c6ac8d53a441ea9dcdaf99a529d5fe577a01748dcee0a5dd267a5ca105a4f37f" Oct 31 03:25:13.214166 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:13.214141 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b"] Oct 31 03:25:13.220266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:13.220241 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2206e7-74445cdc6c-q6v6b"] Oct 31 03:25:14.436800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:14.436766 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6986cbad-3cdf-4333-b1ab-d49bc8bc3eea" path="/var/lib/kubelet/pods/6986cbad-3cdf-4333-b1ab-d49bc8bc3eea/volumes" Oct 31 03:25:16.237285 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:16.237231 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp"] Oct 31 03:25:16.237760 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:16.237504 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" containerID="cri-o://37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa" gracePeriod=30 Oct 31 03:25:16.243172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:16.243145 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57"] Oct 31 03:25:16.243679 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:16.243646 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" containerID="cri-o://6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a" gracePeriod=30 Oct 31 03:25:18.761795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:18.761763 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq"] Oct 31 03:25:18.762939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:18.762899 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" containerID="cri-o://6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab" gracePeriod=30 Oct 31 03:25:18.862993 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:18.862953 4456 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:25:18.863206 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:18.863021 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:25:19.284847 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.284812 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp"] Oct 31 03:25:19.285339 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.285325 4456 cpu_manager.go:414] "containerMap: removing container" podUID="6986cbad-3cdf-4333-b1ab-d49bc8bc3eea" containerName="payload-pre-processing" Oct 31 03:25:19.285396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.285340 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6986cbad-3cdf-4333-b1ab-d49bc8bc3eea" containerName="payload-pre-processing" Oct 31 03:25:19.285396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.285368 4456 cpu_manager.go:414] "containerMap: removing container" podUID="9fd297f3-62d3-4fc8-81ad-7973c85a402e" containerName="payload-pre-processing" Oct 31 03:25:19.285396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.285373 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd297f3-62d3-4fc8-81ad-7973c85a402e" containerName="payload-pre-processing" Oct 31 03:25:19.285396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.285396 4456 cpu_manager.go:414] "containerMap: removing container" podUID="4438eea7-a68f-4d92-9328-9c45998fb8b0" containerName="revoke-keys" Oct 31 03:25:19.285522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.285401 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4438eea7-a68f-4d92-9328-9c45998fb8b0" containerName="revoke-keys" Oct 31 03:25:19.285522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.285467 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd297f3-62d3-4fc8-81ad-7973c85a402e" containerName="payload-pre-processing" Oct 31 03:25:19.285522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.285480 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="4438eea7-a68f-4d92-9328-9c45998fb8b0" containerName="revoke-keys" Oct 31 03:25:19.285522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.285488 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6986cbad-3cdf-4333-b1ab-d49bc8bc3eea" containerName="payload-pre-processing" Oct 31 03:25:19.290327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.290301 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.298341 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.298310 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp"] Oct 31 03:25:19.370883 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.370752 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:25:19.370883 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.370807 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:25:19.386089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.384154 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6r2h\" (UniqueName: \"kubernetes.io/projected/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-kube-api-access-c6r2h\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.386089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.384250 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.386089 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.384386 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.485302 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.485266 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.485591 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.485569 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.485698 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.485673 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6r2h\" (UniqueName: \"kubernetes.io/projected/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-kube-api-access-c6r2h\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.486285 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.486263 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.487726 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.487708 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.506081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.500474 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6r2h\" (UniqueName: \"kubernetes.io/projected/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-kube-api-access-c6r2h\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.612120 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.609296 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:19.760061 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.754309 4456 scope.go:122] "RemoveContainer" containerID="dd8a13ceeee62949dd6b95c3b1c3a6e44e21ba6a3d6e4e88ad7468db0f72b341" Oct 31 03:25:19.772135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.771949 4456 scope.go:122] "RemoveContainer" containerID="12d9a24bf845dc5d32d994d10a7afcbbdc4e1e7d8df96796102ca018334bf39d" Oct 31 03:25:19.942373 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:25:19.942343 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a49f6c9_707f_4c21_ba46_4fb65fd6619f.slice/crio-93ee09cad8c05618aa237574361e453220b1debb591530c2dd80bce280b49cb8 WatchSource:0}: Error finding container 93ee09cad8c05618aa237574361e453220b1debb591530c2dd80bce280b49cb8: Status 404 returned error can't find the container with id 93ee09cad8c05618aa237574361e453220b1debb591530c2dd80bce280b49cb8 Oct 31 03:25:19.944477 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:19.944446 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp"] Oct 31 03:25:20.265259 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:20.265225 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" event={"ID":"3a49f6c9-707f-4c21-ba46-4fb65fd6619f","Type":"ContainerStarted","Data":"93ee09cad8c05618aa237574361e453220b1debb591530c2dd80bce280b49cb8"} Oct 31 03:25:21.278342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.278308 4456 generic.go:356] "Generic (PLEG): container finished" podID="3a49f6c9-707f-4c21-ba46-4fb65fd6619f" containerID="82aff105cea612310a59abeca229de21d7820c68639af9a206b2f11e27f2173e" exitCode=0 Oct 31 03:25:21.278668 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.278409 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" event={"ID":"3a49f6c9-707f-4c21-ba46-4fb65fd6619f","Type":"ContainerDied","Data":"82aff105cea612310a59abeca229de21d7820c68639af9a206b2f11e27f2173e"} Oct 31 03:25:21.550795 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.550751 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:25:21.585467 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.585440 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:25:21.631992 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.631958 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-envoy\") pod \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " Oct 31 03:25:21.631992 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.631995 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-credential-socket\") pod \"5e8cbfb8-a275-4705-9cea-441b590f2436\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " Oct 31 03:25:21.632274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632025 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-socket\") pod \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " Oct 31 03:25:21.632274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632070 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-certs\") pod \"5e8cbfb8-a275-4705-9cea-441b590f2436\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " Oct 31 03:25:21.632274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632109 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-data\") pod \"5e8cbfb8-a275-4705-9cea-441b590f2436\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " Oct 31 03:25:21.632274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632170 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-token\" (UniqueName: \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-token\") pod \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " Oct 31 03:25:21.632561 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632367 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-socket" pod "371d1f98-d538-49f5-9e4e-04cbb3739b4b" (UID: "371d1f98-d538-49f5-9e4e-04cbb3739b4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.632561 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632379 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-certs" pod "5e8cbfb8-a275-4705-9cea-441b590f2436" (UID: "5e8cbfb8-a275-4705-9cea-441b590f2436"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.632561 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632459 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-data" pod "5e8cbfb8-a275-4705-9cea-441b590f2436" (UID: "5e8cbfb8-a275-4705-9cea-441b590f2436"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.632742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632634 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-credential-socket\") pod \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " Oct 31 03:25:21.632742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632680 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-credential-socket" pod "371d1f98-d538-49f5-9e4e-04cbb3739b4b" (UID: "371d1f98-d538-49f5-9e4e-04cbb3739b4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.632742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632704 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-envoy\") pod \"5e8cbfb8-a275-4705-9cea-441b590f2436\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " Oct 31 03:25:21.632742 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632735 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-kube-api-access-6wvnp\" (UniqueName: \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-kube-api-access-6wvnp\") pod \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " Oct 31 03:25:21.633003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632773 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-token\" (UniqueName: \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-token\") pod \"5e8cbfb8-a275-4705-9cea-441b590f2436\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " Oct 31 03:25:21.633003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632811 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-socket\") pod \"5e8cbfb8-a275-4705-9cea-441b590f2436\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " Oct 31 03:25:21.633003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632848 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-podinfo\") pod \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " Oct 31 03:25:21.633003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632879 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-kube-api-access-nqtw4\" (UniqueName: \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-kube-api-access-nqtw4\") pod \"5e8cbfb8-a275-4705-9cea-441b590f2436\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " Oct 31 03:25:21.633003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632914 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istiod-ca-cert\") pod \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " Oct 31 03:25:21.633003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.632967 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8cbfb8-a275-4705-9cea-441b590f2436-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8cbfb8-a275-4705-9cea-441b590f2436-istiod-ca-cert\") pod \"5e8cbfb8-a275-4705-9cea-441b590f2436\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " Oct 31 03:25:21.633356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.633063 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-data\") pod \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " Oct 31 03:25:21.633356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.633115 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-podinfo\") pod \"5e8cbfb8-a275-4705-9cea-441b590f2436\" (UID: \"5e8cbfb8-a275-4705-9cea-441b590f2436\") " Oct 31 03:25:21.633356 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.633162 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-certs\") pod \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\" (UID: \"371d1f98-d538-49f5-9e4e-04cbb3739b4b\") " Oct 31 03:25:21.633971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.633619 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.633971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.633646 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.633971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.633663 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.633971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.633677 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.633971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.633859 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-certs" pod "371d1f98-d538-49f5-9e4e-04cbb3739b4b" (UID: "371d1f98-d538-49f5-9e4e-04cbb3739b4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.633971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.633932 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istiod-ca-cert" pod "371d1f98-d538-49f5-9e4e-04cbb3739b4b" (UID: "371d1f98-d538-49f5-9e4e-04cbb3739b4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:21.634441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.634058 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-data" pod "371d1f98-d538-49f5-9e4e-04cbb3739b4b" (UID: "371d1f98-d538-49f5-9e4e-04cbb3739b4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.634441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.634224 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8cbfb8-a275-4705-9cea-441b590f2436-istiod-ca-cert" pod "5e8cbfb8-a275-4705-9cea-441b590f2436" (UID: "5e8cbfb8-a275-4705-9cea-441b590f2436"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:21.634441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.634276 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-socket" pod "5e8cbfb8-a275-4705-9cea-441b590f2436" (UID: "5e8cbfb8-a275-4705-9cea-441b590f2436"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.634441 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.634386 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-credential-socket" pod "5e8cbfb8-a275-4705-9cea-441b590f2436" (UID: "5e8cbfb8-a275-4705-9cea-441b590f2436"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.634785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.634758 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-envoy" pod "371d1f98-d538-49f5-9e4e-04cbb3739b4b" (UID: "371d1f98-d538-49f5-9e4e-04cbb3739b4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.634911 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.634884 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-token" pod "371d1f98-d538-49f5-9e4e-04cbb3739b4b" (UID: "371d1f98-d538-49f5-9e4e-04cbb3739b4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:21.636402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.636378 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-podinfo" pod "371d1f98-d538-49f5-9e4e-04cbb3739b4b" (UID: "371d1f98-d538-49f5-9e4e-04cbb3739b4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:21.636772 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.636751 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-kube-api-access-nqtw4" pod "5e8cbfb8-a275-4705-9cea-441b590f2436" (UID: "5e8cbfb8-a275-4705-9cea-441b590f2436"). InnerVolumeSpecName "kube-api-access-nqtw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:21.637025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.637003 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-podinfo" pod "5e8cbfb8-a275-4705-9cea-441b590f2436" (UID: "5e8cbfb8-a275-4705-9cea-441b590f2436"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:21.637156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.637124 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-token" pod "5e8cbfb8-a275-4705-9cea-441b590f2436" (UID: "5e8cbfb8-a275-4705-9cea-441b590f2436"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:21.637328 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.637301 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-envoy" pod "5e8cbfb8-a275-4705-9cea-441b590f2436" (UID: "5e8cbfb8-a275-4705-9cea-441b590f2436"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.638244 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.638224 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-kube-api-access-6wvnp" pod "371d1f98-d538-49f5-9e4e-04cbb3739b4b" (UID: "371d1f98-d538-49f5-9e4e-04cbb3739b4b"). InnerVolumeSpecName "kube-api-access-6wvnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:21.734951 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.734905 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.734951 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.734936 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.734951 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.734946 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.734951 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.734956 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.734951 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.734964 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.735364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.734974 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.735364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.734982 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.735364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.734991 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wvnp\" (UniqueName: \"kubernetes.io/projected/371d1f98-d538-49f5-9e4e-04cbb3739b4b-kube-api-access-6wvnp\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.735364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.735000 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.735364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.735009 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cbfb8-a275-4705-9cea-441b590f2436-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.735364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.735017 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.735364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.735027 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqtw4\" (UniqueName: \"kubernetes.io/projected/5e8cbfb8-a275-4705-9cea-441b590f2436-kube-api-access-nqtw4\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.735364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.735050 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371d1f98-d538-49f5-9e4e-04cbb3739b4b-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.735364 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:21.735059 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8cbfb8-a275-4705-9cea-441b590f2436-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:22.288227 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.288184 4456 generic.go:356] "Generic (PLEG): container finished" podID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerID="6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a" exitCode=0 Oct 31 03:25:22.288724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.288250 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" Oct 31 03:25:22.288724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.288270 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" event={"ID":"5e8cbfb8-a275-4705-9cea-441b590f2436","Type":"ContainerDied","Data":"6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a"} Oct 31 03:25:22.288724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.288344 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57" event={"ID":"5e8cbfb8-a275-4705-9cea-441b590f2436","Type":"ContainerDied","Data":"91c90a193dab0d999fd18f092c450a63501389799ccccaf557a00c063e259b2e"} Oct 31 03:25:22.288724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.288376 4456 scope.go:122] "RemoveContainer" containerID="6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a" Oct 31 03:25:22.290576 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.290552 4456 generic.go:356] "Generic (PLEG): container finished" podID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerID="37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa" exitCode=0 Oct 31 03:25:22.290808 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.290791 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" Oct 31 03:25:22.293028 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.292968 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" event={"ID":"371d1f98-d538-49f5-9e4e-04cbb3739b4b","Type":"ContainerDied","Data":"37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa"} Oct 31 03:25:22.293028 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.293020 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp" event={"ID":"371d1f98-d538-49f5-9e4e-04cbb3739b4b","Type":"ContainerDied","Data":"f2b94841a43486fb2a15ce2ae86ff6fab2e21c3022e643331a2ebee40e04813f"} Oct 31 03:25:22.302946 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.302898 4456 scope.go:122] "RemoveContainer" containerID="6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a" Oct 31 03:25:22.303355 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:25:22.303325 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a\": container with ID starting with 6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a not found: ID does not exist" containerID="6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a" Oct 31 03:25:22.303423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.303366 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a"} err="failed to get container status \"6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a\": rpc error: code = NotFound desc = could not find container \"6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a\": container with ID starting with 6b9f96086c0464c57deb242a91b5ba4de9c6b2fddeaaea2ababa1a348fa7599a not found: ID does not exist" Oct 31 03:25:22.303423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.303390 4456 scope.go:122] "RemoveContainer" containerID="37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa" Oct 31 03:25:22.314919 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.314902 4456 scope.go:122] "RemoveContainer" containerID="37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa" Oct 31 03:25:22.315301 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:25:22.315276 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa\": container with ID starting with 37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa not found: ID does not exist" containerID="37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa" Oct 31 03:25:22.315378 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.315307 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa"} err="failed to get container status \"37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa\": rpc error: code = NotFound desc = could not find container \"37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa\": container with ID starting with 37c22199baf65eab2f5eb124489d817f65b3101612d0cde9f27cc9f91ef3fdfa not found: ID does not exist" Oct 31 03:25:22.320627 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.320574 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp"] Oct 31 03:25:22.327785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.327760 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-np8xp"] Oct 31 03:25:22.342465 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.342437 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57"] Oct 31 03:25:22.345761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.345736 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-bsm57"] Oct 31 03:25:22.435067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.435022 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" path="/var/lib/kubelet/pods/371d1f98-d538-49f5-9e4e-04cbb3739b4b/volumes" Oct 31 03:25:22.435572 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.435552 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:22.435629 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.435607 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" path="/var/lib/kubelet/pods/5e8cbfb8-a275-4705-9cea-441b590f2436/volumes" Oct 31 03:25:22.545611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.545535 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-xks-ca\") pod \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " Oct 31 03:25:22.545611 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.545602 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-openshift-ca\") pod \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " Oct 31 03:25:22.545773 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.545719 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-kube-api-access-c6r2h\" (UniqueName: \"kubernetes.io/projected/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-kube-api-access-c6r2h\") pod \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\" (UID: \"3a49f6c9-707f-4c21-ba46-4fb65fd6619f\") " Oct 31 03:25:22.545960 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.545938 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-openshift-ca" pod "3a49f6c9-707f-4c21-ba46-4fb65fd6619f" (UID: "3a49f6c9-707f-4c21-ba46-4fb65fd6619f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:22.546225 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.546194 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:22.547796 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.547770 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-kube-api-access-c6r2h" pod "3a49f6c9-707f-4c21-ba46-4fb65fd6619f" (UID: "3a49f6c9-707f-4c21-ba46-4fb65fd6619f"). InnerVolumeSpecName "kube-api-access-c6r2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:22.547954 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.547787 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-xks-ca" pod "3a49f6c9-707f-4c21-ba46-4fb65fd6619f" (UID: "3a49f6c9-707f-4c21-ba46-4fb65fd6619f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:22.647756 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.647717 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6r2h\" (UniqueName: \"kubernetes.io/projected/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-kube-api-access-c6r2h\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:22.647756 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:22.647757 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a49f6c9-707f-4c21-ba46-4fb65fd6619f-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:23.303524 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:23.303479 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" event={"ID":"3a49f6c9-707f-4c21-ba46-4fb65fd6619f","Type":"ContainerDied","Data":"93ee09cad8c05618aa237574361e453220b1debb591530c2dd80bce280b49cb8"} Oct 31 03:25:23.304144 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:23.303587 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93ee09cad8c05618aa237574361e453220b1debb591530c2dd80bce280b49cb8" Oct 31 03:25:23.304144 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:23.303678 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp" Oct 31 03:25:24.031789 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.031732 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:25:24.163775 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.163737 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-credential-socket\") pod \"be2dcbdd-092e-492e-982b-0b7c070503f4\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " Oct 31 03:25:24.163954 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.163800 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-kube-api-access-x6bf6\" (UniqueName: \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-kube-api-access-x6bf6\") pod \"be2dcbdd-092e-492e-982b-0b7c070503f4\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " Oct 31 03:25:24.163954 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.163887 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-podinfo\") pod \"be2dcbdd-092e-492e-982b-0b7c070503f4\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " Oct 31 03:25:24.164197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.163956 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-envoy\") pod \"be2dcbdd-092e-492e-982b-0b7c070503f4\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " Oct 31 03:25:24.164197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164059 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-credential-socket" pod "be2dcbdd-092e-492e-982b-0b7c070503f4" (UID: "be2dcbdd-092e-492e-982b-0b7c070503f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.164197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164074 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-socket\") pod \"be2dcbdd-092e-492e-982b-0b7c070503f4\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " Oct 31 03:25:24.164197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164118 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2dcbdd-092e-492e-982b-0b7c070503f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2dcbdd-092e-492e-982b-0b7c070503f4-istiod-ca-cert\") pod \"be2dcbdd-092e-492e-982b-0b7c070503f4\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " Oct 31 03:25:24.164197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164161 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-certs\") pod \"be2dcbdd-092e-492e-982b-0b7c070503f4\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " Oct 31 03:25:24.164503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164228 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-data\") pod \"be2dcbdd-092e-492e-982b-0b7c070503f4\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " Oct 31 03:25:24.164503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164278 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-token\" (UniqueName: \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-token\") pod \"be2dcbdd-092e-492e-982b-0b7c070503f4\" (UID: \"be2dcbdd-092e-492e-982b-0b7c070503f4\") " Oct 31 03:25:24.164503 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164357 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-socket" pod "be2dcbdd-092e-492e-982b-0b7c070503f4" (UID: "be2dcbdd-092e-492e-982b-0b7c070503f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.164675 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164579 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-certs" pod "be2dcbdd-092e-492e-982b-0b7c070503f4" (UID: "be2dcbdd-092e-492e-982b-0b7c070503f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.164675 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164631 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2dcbdd-092e-492e-982b-0b7c070503f4-istiod-ca-cert" pod "be2dcbdd-092e-492e-982b-0b7c070503f4" (UID: "be2dcbdd-092e-492e-982b-0b7c070503f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:24.164793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164729 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.164793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164751 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.164793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164773 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2dcbdd-092e-492e-982b-0b7c070503f4-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.164793 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164788 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.165008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.164921 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-data" pod "be2dcbdd-092e-492e-982b-0b7c070503f4" (UID: "be2dcbdd-092e-492e-982b-0b7c070503f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.166824 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.166796 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-envoy" pod "be2dcbdd-092e-492e-982b-0b7c070503f4" (UID: "be2dcbdd-092e-492e-982b-0b7c070503f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.166953 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.166876 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-podinfo" pod "be2dcbdd-092e-492e-982b-0b7c070503f4" (UID: "be2dcbdd-092e-492e-982b-0b7c070503f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:24.166953 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.166950 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-token" pod "be2dcbdd-092e-492e-982b-0b7c070503f4" (UID: "be2dcbdd-092e-492e-982b-0b7c070503f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:24.167126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.166991 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-kube-api-access-x6bf6" pod "be2dcbdd-092e-492e-982b-0b7c070503f4" (UID: "be2dcbdd-092e-492e-982b-0b7c070503f4"). InnerVolumeSpecName "kube-api-access-x6bf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:24.266075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.266028 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.266075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.266068 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.266075 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.266078 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6bf6\" (UniqueName: \"kubernetes.io/projected/be2dcbdd-092e-492e-982b-0b7c070503f4-kube-api-access-x6bf6\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.266403 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.266089 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.266403 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.266099 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2dcbdd-092e-492e-982b-0b7c070503f4-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.323510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.322714 4456 generic.go:356] "Generic (PLEG): container finished" podID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerID="6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab" exitCode=0 Oct 31 03:25:24.323510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.322807 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" event={"ID":"be2dcbdd-092e-492e-982b-0b7c070503f4","Type":"ContainerDied","Data":"6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab"} Oct 31 03:25:24.323510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.322862 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" event={"ID":"be2dcbdd-092e-492e-982b-0b7c070503f4","Type":"ContainerDied","Data":"b1ebc320b43643173229bb9f249d8416893eecb1b4ada641501c81793e2b8fc9"} Oct 31 03:25:24.323510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.322896 4456 scope.go:122] "RemoveContainer" containerID="6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab" Oct 31 03:25:24.323510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.323297 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq" Oct 31 03:25:24.336696 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.336676 4456 scope.go:122] "RemoveContainer" containerID="6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab" Oct 31 03:25:24.336998 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:25:24.336978 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab\": container with ID starting with 6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab not found: ID does not exist" containerID="6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab" Oct 31 03:25:24.337099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.337012 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab"} err="failed to get container status \"6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab\": rpc error: code = NotFound desc = could not find container \"6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab\": container with ID starting with 6e534b52811a15e7c9ac918a20019ef5d660c3253422c27661fd42bf624fdbab not found: ID does not exist" Oct 31 03:25:24.352917 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.352884 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq"] Oct 31 03:25:24.358572 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.358549 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-swxrq"] Oct 31 03:25:24.437861 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:24.437829 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" path="/var/lib/kubelet/pods/be2dcbdd-092e-492e-982b-0b7c070503f4/volumes" Oct 31 03:25:29.383111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:29.380199 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4"] Oct 31 03:25:29.385852 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:29.385583 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg"] Oct 31 03:25:30.967723 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:30.967683 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp"] Oct 31 03:25:30.971180 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:30.971151 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0a531d-s9jwp"] Oct 31 03:25:32.134774 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:25:32.134734 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:25:32.435905 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:32.435875 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a49f6c9-707f-4c21-ba46-4fb65fd6619f" path="/var/lib/kubelet/pods/3a49f6c9-707f-4c21-ba46-4fb65fd6619f/volumes" Oct 31 03:25:34.380808 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.380760 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" podUID="1f819a08-2680-493d-80a4-d84b281d78e4" containerName="payload-processing" containerID="cri-o://275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c" gracePeriod=25 Oct 31 03:25:34.387382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.387344 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" podUID="72ebb1b0-8737-4275-8c09-5d5152251c81" containerName="payload-pre-processing" containerID="cri-o://8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21" gracePeriod=25 Oct 31 03:25:34.658749 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.658712 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:25:34.699430 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.699396 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:25:34.780031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.779989 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f819a08-2680-493d-80a4-d84b281d78e4-kube-api-access-s7knx\" (UniqueName: \"kubernetes.io/projected/1f819a08-2680-493d-80a4-d84b281d78e4-kube-api-access-s7knx\") pod \"1f819a08-2680-493d-80a4-d84b281d78e4\" (UID: \"1f819a08-2680-493d-80a4-d84b281d78e4\") " Oct 31 03:25:34.780260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.780118 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f819a08-2680-493d-80a4-d84b281d78e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f819a08-2680-493d-80a4-d84b281d78e4-plugins-config-volume\") pod \"1f819a08-2680-493d-80a4-d84b281d78e4\" (UID: \"1f819a08-2680-493d-80a4-d84b281d78e4\") " Oct 31 03:25:34.780502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.780481 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f819a08-2680-493d-80a4-d84b281d78e4-plugins-config-volume" pod "1f819a08-2680-493d-80a4-d84b281d78e4" (UID: "1f819a08-2680-493d-80a4-d84b281d78e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:34.782207 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.782176 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f819a08-2680-493d-80a4-d84b281d78e4-kube-api-access-s7knx" pod "1f819a08-2680-493d-80a4-d84b281d78e4" (UID: "1f819a08-2680-493d-80a4-d84b281d78e4"). InnerVolumeSpecName "kube-api-access-s7knx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:34.880911 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.880870 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ebb1b0-8737-4275-8c09-5d5152251c81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ebb1b0-8737-4275-8c09-5d5152251c81-plugins-config-volume\") pod \"72ebb1b0-8737-4275-8c09-5d5152251c81\" (UID: \"72ebb1b0-8737-4275-8c09-5d5152251c81\") " Oct 31 03:25:34.881131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.880976 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ebb1b0-8737-4275-8c09-5d5152251c81-kube-api-access-nm9qv\" (UniqueName: \"kubernetes.io/projected/72ebb1b0-8737-4275-8c09-5d5152251c81-kube-api-access-nm9qv\") pod \"72ebb1b0-8737-4275-8c09-5d5152251c81\" (UID: \"72ebb1b0-8737-4275-8c09-5d5152251c81\") " Oct 31 03:25:34.881254 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.881239 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f819a08-2680-493d-80a4-d84b281d78e4-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:34.881297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.881257 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7knx\" (UniqueName: \"kubernetes.io/projected/1f819a08-2680-493d-80a4-d84b281d78e4-kube-api-access-s7knx\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:34.881297 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.881268 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ebb1b0-8737-4275-8c09-5d5152251c81-plugins-config-volume" pod "72ebb1b0-8737-4275-8c09-5d5152251c81" (UID: "72ebb1b0-8737-4275-8c09-5d5152251c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:34.883109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.883088 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ebb1b0-8737-4275-8c09-5d5152251c81-kube-api-access-nm9qv" pod "72ebb1b0-8737-4275-8c09-5d5152251c81" (UID: "72ebb1b0-8737-4275-8c09-5d5152251c81"). InnerVolumeSpecName "kube-api-access-nm9qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:34.982700 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.982659 4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ebb1b0-8737-4275-8c09-5d5152251c81-plugins-config-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:34.982700 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:34.982692 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm9qv\" (UniqueName: \"kubernetes.io/projected/72ebb1b0-8737-4275-8c09-5d5152251c81-kube-api-access-nm9qv\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:35.414084 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.414017 4456 generic.go:356] "Generic (PLEG): container finished" podID="1f819a08-2680-493d-80a4-d84b281d78e4" containerID="275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c" exitCode=0 Oct 31 03:25:35.414519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.414101 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" Oct 31 03:25:35.414519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.414109 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" event={"ID":"1f819a08-2680-493d-80a4-d84b281d78e4","Type":"ContainerDied","Data":"275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c"} Oct 31 03:25:35.414519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.414176 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4" event={"ID":"1f819a08-2680-493d-80a4-d84b281d78e4","Type":"ContainerDied","Data":"94cff9bd7d5a784951db96207e55306149ace1d8433bbc35984d8983c616bd7d"} Oct 31 03:25:35.414519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.414208 4456 scope.go:122] "RemoveContainer" containerID="275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c" Oct 31 03:25:35.415951 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.415921 4456 generic.go:356] "Generic (PLEG): container finished" podID="72ebb1b0-8737-4275-8c09-5d5152251c81" containerID="8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21" exitCode=0 Oct 31 03:25:35.416128 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.415974 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" Oct 31 03:25:35.416128 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.415979 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" event={"ID":"72ebb1b0-8737-4275-8c09-5d5152251c81","Type":"ContainerDied","Data":"8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21"} Oct 31 03:25:35.416128 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.416059 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg" event={"ID":"72ebb1b0-8737-4275-8c09-5d5152251c81","Type":"ContainerDied","Data":"858079f5c21aed2be3c8e42534b53065adaf7862cf95a93df44541a7b713f00e"} Oct 31 03:25:35.427014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.426997 4456 scope.go:122] "RemoveContainer" containerID="275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c" Oct 31 03:25:35.427352 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:25:35.427329 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c\": container with ID starting with 275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c not found: ID does not exist" containerID="275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c" Oct 31 03:25:35.427422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.427358 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c"} err="failed to get container status \"275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c\": rpc error: code = NotFound desc = could not find container \"275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c\": container with ID starting with 275f92e4a9557ac5365de307e3d0e8019b0ff73f429ef8d86c50feb03b67c48c not found: ID does not exist" Oct 31 03:25:35.427422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.427373 4456 scope.go:122] "RemoveContainer" containerID="8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21" Oct 31 03:25:35.440221 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.440001 4456 scope.go:122] "RemoveContainer" containerID="8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21" Oct 31 03:25:35.449340 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:25:35.441059 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21\": container with ID starting with 8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21 not found: ID does not exist" containerID="8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21" Oct 31 03:25:35.449340 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.441462 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21"} err="failed to get container status \"8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21\": rpc error: code = NotFound desc = could not find container \"8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21\": container with ID starting with 8294e252e7343f2657e8a6794d64883c1295aaddd6bfc276cc16c52594750b21 not found: ID does not exist" Oct 31 03:25:35.455009 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.454983 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4"] Oct 31 03:25:35.459685 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.459663 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0a531d-5dbbd79d95-jf2w4"] Oct 31 03:25:35.472929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.472904 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg"] Oct 31 03:25:35.476100 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:35.476077 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0a531d-dcb4bcb5f-mxnlg"] Oct 31 03:25:36.434087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:36.434046 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f819a08-2680-493d-80a4-d84b281d78e4" path="/var/lib/kubelet/pods/1f819a08-2680-493d-80a4-d84b281d78e4/volumes" Oct 31 03:25:36.434657 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:36.434637 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ebb1b0-8737-4275-8c09-5d5152251c81" path="/var/lib/kubelet/pods/72ebb1b0-8737-4275-8c09-5d5152251c81/volumes" Oct 31 03:25:44.425800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:44.425768 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm"] Oct 31 03:25:50.837014 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.836971 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j"] Oct 31 03:25:50.837659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837637 4456 cpu_manager.go:414] "containerMap: removing container" podUID="3a49f6c9-707f-4c21-ba46-4fb65fd6619f" containerName="revoke-keys" Oct 31 03:25:50.837761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837661 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a49f6c9-707f-4c21-ba46-4fb65fd6619f" containerName="revoke-keys" Oct 31 03:25:50.837761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837681 4456 cpu_manager.go:414] "containerMap: removing container" podUID="72ebb1b0-8737-4275-8c09-5d5152251c81" containerName="payload-pre-processing" Oct 31 03:25:50.837761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837688 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ebb1b0-8737-4275-8c09-5d5152251c81" containerName="payload-pre-processing" Oct 31 03:25:50.837761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837726 4456 cpu_manager.go:414] "containerMap: removing container" podUID="1f819a08-2680-493d-80a4-d84b281d78e4" containerName="payload-processing" Oct 31 03:25:50.837761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837734 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f819a08-2680-493d-80a4-d84b281d78e4" containerName="payload-processing" Oct 31 03:25:50.837761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837747 4456 cpu_manager.go:414] "containerMap: removing container" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" Oct 31 03:25:50.837761 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837755 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" Oct 31 03:25:50.838172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837787 4456 cpu_manager.go:414] "containerMap: removing container" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" Oct 31 03:25:50.838172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837795 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" Oct 31 03:25:50.838172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837805 4456 cpu_manager.go:414] "containerMap: removing container" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" Oct 31 03:25:50.838172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837812 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" Oct 31 03:25:50.838172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837926 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="371d1f98-d538-49f5-9e4e-04cbb3739b4b" containerName="istio-proxy" Oct 31 03:25:50.838172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837941 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8cbfb8-a275-4705-9cea-441b590f2436" containerName="istio-proxy" Oct 31 03:25:50.838172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837954 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a49f6c9-707f-4c21-ba46-4fb65fd6619f" containerName="revoke-keys" Oct 31 03:25:50.838172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837964 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ebb1b0-8737-4275-8c09-5d5152251c81" containerName="payload-pre-processing" Oct 31 03:25:50.838172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837977 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f819a08-2680-493d-80a4-d84b281d78e4" containerName="payload-processing" Oct 31 03:25:50.838172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.837988 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2dcbdd-092e-492e-982b-0b7c070503f4" containerName="istio-proxy" Oct 31 03:25:50.841725 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.841704 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:50.854431 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.854401 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j"] Oct 31 03:25:50.939938 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.939899 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:25:50.940131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.940108 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:25:50.953619 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.953590 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhkzt\" (UniqueName: \"kubernetes.io/projected/8aae1fe9-9261-421b-b057-80ad4a8fbdef-kube-api-access-dhkzt\") pod \"maas-api-revoke-sub-ed261c50-4h29j\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:50.953800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.953641 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed261c50-4h29j\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:50.953800 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:50.953754 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed261c50-4h29j\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:51.054665 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:51.054623 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhkzt\" (UniqueName: \"kubernetes.io/projected/8aae1fe9-9261-421b-b057-80ad4a8fbdef-kube-api-access-dhkzt\") pod \"maas-api-revoke-sub-ed261c50-4h29j\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:51.054835 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:51.054677 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed261c50-4h29j\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:51.054835 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:51.054728 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed261c50-4h29j\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:51.055420 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:51.055398 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed261c50-4h29j\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:51.057315 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:51.057288 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed261c50-4h29j\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:51.068165 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:51.068141 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhkzt\" (UniqueName: \"kubernetes.io/projected/8aae1fe9-9261-421b-b057-80ad4a8fbdef-kube-api-access-dhkzt\") pod \"maas-api-revoke-sub-ed261c50-4h29j\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:51.163031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:51.163001 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:51.445054 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:25:51.445007 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aae1fe9_9261_421b_b057_80ad4a8fbdef.slice/crio-6c092aa3e947623806130f3a9670b766f2f54077abaa74b64a254a390cf27719 WatchSource:0}: Error finding container 6c092aa3e947623806130f3a9670b766f2f54077abaa74b64a254a390cf27719: Status 404 returned error can't find the container with id 6c092aa3e947623806130f3a9670b766f2f54077abaa74b64a254a390cf27719 Oct 31 03:25:51.446714 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:51.446682 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j"] Oct 31 03:25:51.556887 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:51.556846 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" event={"ID":"8aae1fe9-9261-421b-b057-80ad4a8fbdef","Type":"ContainerStarted","Data":"6c092aa3e947623806130f3a9670b766f2f54077abaa74b64a254a390cf27719"} Oct 31 03:25:52.567430 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:52.567394 4456 generic.go:356] "Generic (PLEG): container finished" podID="8aae1fe9-9261-421b-b057-80ad4a8fbdef" containerID="e03a98c2413639f65184b9fdd6905a4f61650261fe35958abca84ff285a0781a" exitCode=0 Oct 31 03:25:52.567840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:52.567439 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" event={"ID":"8aae1fe9-9261-421b-b057-80ad4a8fbdef","Type":"ContainerDied","Data":"e03a98c2413639f65184b9fdd6905a4f61650261fe35958abca84ff285a0781a"} Oct 31 03:25:53.720510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:53.720482 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:53.785222 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:53.785189 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aae1fe9-9261-421b-b057-80ad4a8fbdef-kube-api-access-dhkzt\" (UniqueName: \"kubernetes.io/projected/8aae1fe9-9261-421b-b057-80ad4a8fbdef-kube-api-access-dhkzt\") pod \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " Oct 31 03:25:53.785378 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:53.785367 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-xks-ca\") pod \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " Oct 31 03:25:53.785421 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:53.785401 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-openshift-ca\") pod \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\" (UID: \"8aae1fe9-9261-421b-b057-80ad4a8fbdef\") " Oct 31 03:25:53.785804 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:53.785781 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-openshift-ca" pod "8aae1fe9-9261-421b-b057-80ad4a8fbdef" (UID: "8aae1fe9-9261-421b-b057-80ad4a8fbdef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:53.787371 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:53.787348 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-xks-ca" pod "8aae1fe9-9261-421b-b057-80ad4a8fbdef" (UID: "8aae1fe9-9261-421b-b057-80ad4a8fbdef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:53.787468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:53.787384 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aae1fe9-9261-421b-b057-80ad4a8fbdef-kube-api-access-dhkzt" pod "8aae1fe9-9261-421b-b057-80ad4a8fbdef" (UID: "8aae1fe9-9261-421b-b057-80ad4a8fbdef"). InnerVolumeSpecName "kube-api-access-dhkzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:53.886683 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:53.886591 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:53.886683 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:53.886626 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aae1fe9-9261-421b-b057-80ad4a8fbdef-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:53.886683 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:53.886641 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhkzt\" (UniqueName: \"kubernetes.io/projected/8aae1fe9-9261-421b-b057-80ad4a8fbdef-kube-api-access-dhkzt\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:54.584420 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:54.584389 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" Oct 31 03:25:54.584624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:54.584422 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j" event={"ID":"8aae1fe9-9261-421b-b057-80ad4a8fbdef","Type":"ContainerDied","Data":"6c092aa3e947623806130f3a9670b766f2f54077abaa74b64a254a390cf27719"} Oct 31 03:25:54.584624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:54.584473 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c092aa3e947623806130f3a9670b766f2f54077abaa74b64a254a390cf27719" Oct 31 03:25:59.448288 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.448249 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" containerID="cri-o://d3928741a641c1eddfc45e12fb9b443476af4aae996032df0ac5e24f2b705975" gracePeriod=45 Oct 31 03:25:59.575762 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.575713 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused" Oct 31 03:25:59.629834 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.629805 4456 generic.go:356] "Generic (PLEG): container finished" podID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerID="d3928741a641c1eddfc45e12fb9b443476af4aae996032df0ac5e24f2b705975" exitCode=0 Oct 31 03:25:59.629992 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.629878 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" event={"ID":"391ef3a2-cff8-481b-898a-b00a84351d8d","Type":"ContainerDied","Data":"d3928741a641c1eddfc45e12fb9b443476af4aae996032df0ac5e24f2b705975"} Oct 31 03:25:59.696366 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.696344 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:25:59.844544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.844455 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391ef3a2-cff8-481b-898a-b00a84351d8d-kube-api-access-bmvmq\" (UniqueName: \"kubernetes.io/projected/391ef3a2-cff8-481b-898a-b00a84351d8d-kube-api-access-bmvmq\") pod \"391ef3a2-cff8-481b-898a-b00a84351d8d\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " Oct 31 03:25:59.844544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.844499 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-tmp-dir\") pod \"391ef3a2-cff8-481b-898a-b00a84351d8d\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " Oct 31 03:25:59.844747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.844569 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-home\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-home\") pod \"391ef3a2-cff8-481b-898a-b00a84351d8d\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " Oct 31 03:25:59.844747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.844611 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-model-cache\") pod \"391ef3a2-cff8-481b-898a-b00a84351d8d\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " Oct 31 03:25:59.844865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.844763 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-dshm\") pod \"391ef3a2-cff8-481b-898a-b00a84351d8d\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " Oct 31 03:25:59.844865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.844810 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-tmp-dir" pod "391ef3a2-cff8-481b-898a-b00a84351d8d" (UID: "391ef3a2-cff8-481b-898a-b00a84351d8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.844987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.844891 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-home" pod "391ef3a2-cff8-481b-898a-b00a84351d8d" (UID: "391ef3a2-cff8-481b-898a-b00a84351d8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.845067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.844998 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-model-cache" pod "391ef3a2-cff8-481b-898a-b00a84351d8d" (UID: "391ef3a2-cff8-481b-898a-b00a84351d8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.845144 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.845118 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391ef3a2-cff8-481b-898a-b00a84351d8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/391ef3a2-cff8-481b-898a-b00a84351d8d-tls-certs\") pod \"391ef3a2-cff8-481b-898a-b00a84351d8d\" (UID: \"391ef3a2-cff8-481b-898a-b00a84351d8d\") " Oct 31 03:25:59.845548 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.845520 4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-tmp-dir\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.845548 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.845549 4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-home\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.845745 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.845566 4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-model-cache\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.846872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.846842 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391ef3a2-cff8-481b-898a-b00a84351d8d-kube-api-access-bmvmq" pod "391ef3a2-cff8-481b-898a-b00a84351d8d" (UID: "391ef3a2-cff8-481b-898a-b00a84351d8d"). InnerVolumeSpecName "kube-api-access-bmvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:59.846965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.846884 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-dshm" pod "391ef3a2-cff8-481b-898a-b00a84351d8d" (UID: "391ef3a2-cff8-481b-898a-b00a84351d8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.847013 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.846980 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391ef3a2-cff8-481b-898a-b00a84351d8d-tls-certs" pod "391ef3a2-cff8-481b-898a-b00a84351d8d" (UID: "391ef3a2-cff8-481b-898a-b00a84351d8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:59.946437 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.946401 4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/391ef3a2-cff8-481b-898a-b00a84351d8d-dshm\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.946437 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.946431 4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391ef3a2-cff8-481b-898a-b00a84351d8d-tls-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.946437 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:25:59.946442 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmvmq\" (UniqueName: \"kubernetes.io/projected/391ef3a2-cff8-481b-898a-b00a84351d8d-kube-api-access-bmvmq\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:00.638601 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:00.638570 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" Oct 31 03:26:00.639133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:00.638606 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm" event={"ID":"391ef3a2-cff8-481b-898a-b00a84351d8d","Type":"ContainerDied","Data":"5afdd26c17b82351d9911c92f6660a8ac5e66b9f43df7290c24cc3aa057a1e70"} Oct 31 03:26:00.639133 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:00.638664 4456 scope.go:122] "RemoveContainer" containerID="d3928741a641c1eddfc45e12fb9b443476af4aae996032df0ac5e24f2b705975" Oct 31 03:26:00.662973 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:00.662928 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm"] Oct 31 03:26:00.666306 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:00.666283 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/simulator-w4-bebea4-kserve-684c9bd584-gkbbm"] Oct 31 03:26:02.134272 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:26:02.134237 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:26:02.434704 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:02.434673 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" path="/var/lib/kubelet/pods/391ef3a2-cff8-481b-898a-b00a84351d8d/volumes" Oct 31 03:26:08.027130 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:08.027088 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9"] Oct 31 03:26:08.030268 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:08.030239 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245c7ba8-ttxg9"] Oct 31 03:26:08.434895 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:08.434852 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a53a6d6-5983-4739-8bb5-5af445c128cd" path="/var/lib/kubelet/pods/4a53a6d6-5983-4739-8bb5-5af445c128cd/volumes" Oct 31 03:26:13.112738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:13.112703 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8"] Oct 31 03:26:13.113229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:13.112954 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" containerID="cri-o://1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e" gracePeriod=30 Oct 31 03:26:13.116658 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:13.116628 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84"] Oct 31 03:26:13.116887 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:13.116842 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" containerID="cri-o://c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869" gracePeriod=30 Oct 31 03:26:15.904809 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:15.904770 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64564ccd46-mqzkl"] Oct 31 03:26:15.905305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:15.905000 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-64564ccd46-mqzkl" podUID="5ecd0946-7571-46bf-9eb7-913be7450dd8" containerName="manager" containerID="cri-o://fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5" gracePeriod=30 Oct 31 03:26:16.172069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.171974 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:26:16.324776 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.324740 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-webhook-cert\") pod \"5ecd0946-7571-46bf-9eb7-913be7450dd8\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " Oct 31 03:26:16.324971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.324876 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-metrics-certs\") pod \"5ecd0946-7571-46bf-9eb7-913be7450dd8\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " Oct 31 03:26:16.324971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.324919 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ecd0946-7571-46bf-9eb7-913be7450dd8-kube-api-access-gh727\" (UniqueName: \"kubernetes.io/projected/5ecd0946-7571-46bf-9eb7-913be7450dd8-kube-api-access-gh727\") pod \"5ecd0946-7571-46bf-9eb7-913be7450dd8\" (UID: \"5ecd0946-7571-46bf-9eb7-913be7450dd8\") " Oct 31 03:26:16.326945 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.326913 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-metrics-certs" pod "5ecd0946-7571-46bf-9eb7-913be7450dd8" (UID: "5ecd0946-7571-46bf-9eb7-913be7450dd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:26:16.327095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.326940 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecd0946-7571-46bf-9eb7-913be7450dd8-kube-api-access-gh727" pod "5ecd0946-7571-46bf-9eb7-913be7450dd8" (UID: "5ecd0946-7571-46bf-9eb7-913be7450dd8"). InnerVolumeSpecName "kube-api-access-gh727". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:16.327095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.326946 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-webhook-cert" pod "5ecd0946-7571-46bf-9eb7-913be7450dd8" (UID: "5ecd0946-7571-46bf-9eb7-913be7450dd8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:26:16.426468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.426385 4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-metrics-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:16.426468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.426415 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh727\" (UniqueName: \"kubernetes.io/projected/5ecd0946-7571-46bf-9eb7-913be7450dd8-kube-api-access-gh727\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:16.426468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.426429 4456 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ecd0946-7571-46bf-9eb7-913be7450dd8-webhook-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:16.772807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.772715 4456 generic.go:356] "Generic (PLEG): container finished" podID="5ecd0946-7571-46bf-9eb7-913be7450dd8" containerID="fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5" exitCode=0 Oct 31 03:26:16.772807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.772771 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64564ccd46-mqzkl" Oct 31 03:26:16.772807 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.772800 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64564ccd46-mqzkl" event={"ID":"5ecd0946-7571-46bf-9eb7-913be7450dd8","Type":"ContainerDied","Data":"fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5"} Oct 31 03:26:16.773158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.772851 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64564ccd46-mqzkl" event={"ID":"5ecd0946-7571-46bf-9eb7-913be7450dd8","Type":"ContainerDied","Data":"7308ded5578e739a7819014b2c04a0ccd54a8720ac62e777c9134706c8723815"} Oct 31 03:26:16.773158 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.772879 4456 scope.go:122] "RemoveContainer" containerID="fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5" Oct 31 03:26:16.786516 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.785905 4456 scope.go:122] "RemoveContainer" containerID="fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5" Oct 31 03:26:16.787275 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:26:16.786639 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5\": container with ID starting with fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5 not found: ID does not exist" containerID="fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5" Oct 31 03:26:16.787275 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.786682 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5"} err="failed to get container status \"fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5\": rpc error: code = NotFound desc = could not find container \"fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5\": container with ID starting with fcbcfc4aae46cf6628a2e700ffd0ce241dbec412091258d7a1727d379387c6d5 not found: ID does not exist" Oct 31 03:26:16.795635 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.795610 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64564ccd46-mqzkl"] Oct 31 03:26:16.799322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:16.799300 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64564ccd46-mqzkl"] Oct 31 03:26:18.524311 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.524274 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ecd0946-7571-46bf-9eb7-913be7450dd8" path="/var/lib/kubelet/pods/5ecd0946-7571-46bf-9eb7-913be7450dd8/volumes" Oct 31 03:26:18.836440 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.836410 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:26:18.838211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.838173 4456 generic.go:356] "Generic (PLEG): container finished" podID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerID="c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869" exitCode=0 Oct 31 03:26:18.838327 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.838286 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" event={"ID":"da341e75-5d33-4a52-b018-bf1b2d2f83e9","Type":"ContainerDied","Data":"c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869"} Oct 31 03:26:18.838412 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.838393 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" event={"ID":"da341e75-5d33-4a52-b018-bf1b2d2f83e9","Type":"ContainerDied","Data":"da275a0c66812cbc60d3238ecef58648b0949f36d3d59e244a30b070bb9fe8b5"} Oct 31 03:26:18.838460 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.838446 4456 scope.go:122] "RemoveContainer" containerID="c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869" Oct 31 03:26:18.839432 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.839401 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:26:18.840258 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.840230 4456 generic.go:356] "Generic (PLEG): container finished" podID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerID="1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e" exitCode=0 Oct 31 03:26:18.840334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.840299 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" event={"ID":"d31b4da9-4b9f-4ff2-85be-343786144bf8","Type":"ContainerDied","Data":"1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e"} Oct 31 03:26:18.840382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.840364 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" event={"ID":"d31b4da9-4b9f-4ff2-85be-343786144bf8","Type":"ContainerDied","Data":"4733deed2ac5f39c118b7aa2ce50c9f0a70180abc71b886fff757f64be90aeb4"} Oct 31 03:26:18.921292 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.921181 4456 scope.go:122] "RemoveContainer" containerID="c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869" Oct 31 03:26:18.921981 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:26:18.921819 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869\": container with ID starting with c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869 not found: ID does not exist" containerID="c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869" Oct 31 03:26:18.921981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.921861 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869"} err="failed to get container status \"c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869\": rpc error: code = NotFound desc = could not find container \"c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869\": container with ID starting with c8487b71a9ac8f26ddd54f05f6150ff2423f231ea4c1838dfb8c031a4df64869 not found: ID does not exist" Oct 31 03:26:18.921981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.921887 4456 scope.go:122] "RemoveContainer" containerID="1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e" Oct 31 03:26:18.928211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.928184 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-data\") pod \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " Oct 31 03:26:18.928385 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.928366 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31b4da9-4b9f-4ff2-85be-343786144bf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31b4da9-4b9f-4ff2-85be-343786144bf8-istiod-ca-cert\") pod \"d31b4da9-4b9f-4ff2-85be-343786144bf8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " Oct 31 03:26:18.928527 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.928510 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-credential-socket\") pod \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " Oct 31 03:26:18.928828 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.928808 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-certs\") pod \"d31b4da9-4b9f-4ff2-85be-343786144bf8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " Oct 31 03:26:18.928987 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.928970 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-certs\") pod \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " Oct 31 03:26:18.929127 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.929110 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-envoy\") pod \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " Oct 31 03:26:18.929308 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.929291 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-kube-api-access-hbzcw\" (UniqueName: \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-kube-api-access-hbzcw\") pod \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " Oct 31 03:26:18.929459 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.929442 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-data\") pod \"d31b4da9-4b9f-4ff2-85be-343786144bf8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " Oct 31 03:26:18.929590 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.929575 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-envoy\") pod \"d31b4da9-4b9f-4ff2-85be-343786144bf8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " Oct 31 03:26:18.929718 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.929701 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-credential-socket\") pod \"d31b4da9-4b9f-4ff2-85be-343786144bf8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " Oct 31 03:26:18.932099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.929835 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istiod-ca-cert\") pod \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " Oct 31 03:26:18.932099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.931119 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-token\" (UniqueName: \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-token\") pod \"d31b4da9-4b9f-4ff2-85be-343786144bf8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " Oct 31 03:26:18.932099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.931144 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-token\" (UniqueName: \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-token\") pod \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " Oct 31 03:26:18.932099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.931195 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-podinfo\") pod \"d31b4da9-4b9f-4ff2-85be-343786144bf8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " Oct 31 03:26:18.932099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.931238 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-podinfo\") pod \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " Oct 31 03:26:18.932099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.931281 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-socket\") pod \"d31b4da9-4b9f-4ff2-85be-343786144bf8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " Oct 31 03:26:18.932099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.931323 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-socket\") pod \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\" (UID: \"da341e75-5d33-4a52-b018-bf1b2d2f83e9\") " Oct 31 03:26:18.932099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.931353 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-kube-api-access-lz4xn\" (UniqueName: \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-kube-api-access-lz4xn\") pod \"d31b4da9-4b9f-4ff2-85be-343786144bf8\" (UID: \"d31b4da9-4b9f-4ff2-85be-343786144bf8\") " Oct 31 03:26:18.935382 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.929612 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-certs" pod "da341e75-5d33-4a52-b018-bf1b2d2f83e9" (UID: "da341e75-5d33-4a52-b018-bf1b2d2f83e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.935485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.929816 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-credential-socket" pod "da341e75-5d33-4a52-b018-bf1b2d2f83e9" (UID: "da341e75-5d33-4a52-b018-bf1b2d2f83e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.935485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.930094 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-certs" pod "d31b4da9-4b9f-4ff2-85be-343786144bf8" (UID: "d31b4da9-4b9f-4ff2-85be-343786144bf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.935485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.930362 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-data" pod "d31b4da9-4b9f-4ff2-85be-343786144bf8" (UID: "d31b4da9-4b9f-4ff2-85be-343786144bf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.935485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.932193 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31b4da9-4b9f-4ff2-85be-343786144bf8-istiod-ca-cert" pod "d31b4da9-4b9f-4ff2-85be-343786144bf8" (UID: "d31b4da9-4b9f-4ff2-85be-343786144bf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:26:18.935485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.933077 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-data" pod "da341e75-5d33-4a52-b018-bf1b2d2f83e9" (UID: "da341e75-5d33-4a52-b018-bf1b2d2f83e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.935485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.934958 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-kube-api-access-lz4xn" pod "d31b4da9-4b9f-4ff2-85be-343786144bf8" (UID: "d31b4da9-4b9f-4ff2-85be-343786144bf8"). InnerVolumeSpecName "kube-api-access-lz4xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:18.935485 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.935346 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-socket" pod "d31b4da9-4b9f-4ff2-85be-343786144bf8" (UID: "d31b4da9-4b9f-4ff2-85be-343786144bf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.938296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.938116 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istiod-ca-cert" pod "da341e75-5d33-4a52-b018-bf1b2d2f83e9" (UID: "da341e75-5d33-4a52-b018-bf1b2d2f83e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:26:18.938296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.938209 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-credential-socket" pod "d31b4da9-4b9f-4ff2-85be-343786144bf8" (UID: "d31b4da9-4b9f-4ff2-85be-343786144bf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.938296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.938273 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-socket" pod "da341e75-5d33-4a52-b018-bf1b2d2f83e9" (UID: "da341e75-5d33-4a52-b018-bf1b2d2f83e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.938502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.938364 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-envoy" pod "da341e75-5d33-4a52-b018-bf1b2d2f83e9" (UID: "da341e75-5d33-4a52-b018-bf1b2d2f83e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.938567 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.938511 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-podinfo" pod "d31b4da9-4b9f-4ff2-85be-343786144bf8" (UID: "d31b4da9-4b9f-4ff2-85be-343786144bf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:26:18.938942 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.938915 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-kube-api-access-hbzcw" pod "da341e75-5d33-4a52-b018-bf1b2d2f83e9" (UID: "da341e75-5d33-4a52-b018-bf1b2d2f83e9"). InnerVolumeSpecName "kube-api-access-hbzcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:18.939085 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.939060 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-token" pod "d31b4da9-4b9f-4ff2-85be-343786144bf8" (UID: "d31b4da9-4b9f-4ff2-85be-343786144bf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:18.939316 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.939293 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-token" pod "da341e75-5d33-4a52-b018-bf1b2d2f83e9" (UID: "da341e75-5d33-4a52-b018-bf1b2d2f83e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:18.940561 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.940534 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-envoy" pod "d31b4da9-4b9f-4ff2-85be-343786144bf8" (UID: "d31b4da9-4b9f-4ff2-85be-343786144bf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.941078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.941029 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-podinfo" pod "da341e75-5d33-4a52-b018-bf1b2d2f83e9" (UID: "da341e75-5d33-4a52-b018-bf1b2d2f83e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:26:18.943076 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.943052 4456 scope.go:122] "RemoveContainer" containerID="1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e" Oct 31 03:26:18.943571 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:26:18.943551 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e\": container with ID starting with 1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e not found: ID does not exist" containerID="1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e" Oct 31 03:26:18.943660 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:18.943583 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e"} err="failed to get container status \"1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e\": rpc error: code = NotFound desc = could not find container \"1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e\": container with ID starting with 1ba37407dc8fb72130b425688f6ca73ca09ceb752506bcf4126b00d5b0c84c1e not found: ID does not exist" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.032947 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033001 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033019 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033059 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033076 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033090 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033104 4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-podinfo\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033121 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033137 4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033152 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz4xn\" (UniqueName: \"kubernetes.io/projected/d31b4da9-4b9f-4ff2-85be-343786144bf8-kube-api-access-lz4xn\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033169 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033183 4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31b4da9-4b9f-4ff2-85be-343786144bf8-istiod-ca-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033197 4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-credential-socket\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033212 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033226 4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-workload-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033240 4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da341e75-5d33-4a52-b018-bf1b2d2f83e9-istio-envoy\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033256 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbzcw\" (UniqueName: \"kubernetes.io/projected/da341e75-5d33-4a52-b018-bf1b2d2f83e9-kube-api-access-hbzcw\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.034307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.033270 4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31b4da9-4b9f-4ff2-85be-343786144bf8-istio-data\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.847724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.847693 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84" Oct 31 03:26:19.848965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.848943 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8" Oct 31 03:26:19.878813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.878771 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84"] Oct 31 03:26:19.881863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.881836 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thr84"] Oct 31 03:26:19.895854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.895829 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8"] Oct 31 03:26:19.902306 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.902285 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-8kfl8"] Oct 31 03:26:19.908610 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.908590 4456 scope.go:122] "RemoveContainer" containerID="6fd3ce937362fd40f6835c214f6324cce9a71fca752341a90a90f11a5c4a9089" Oct 31 03:26:19.918647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.918631 4456 scope.go:122] "RemoveContainer" containerID="3635681c4934fdccb0c70d214e6bff7f64c431f72859ce5408be48efda72cdfb" Oct 31 03:26:19.927858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.927840 4456 scope.go:122] "RemoveContainer" containerID="9e41b1f810d4bf9162180a0f2a88f839ce2ecc6b0d26b01c568819e77fd77699" Oct 31 03:26:19.937642 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.937620 4456 scope.go:122] "RemoveContainer" containerID="da081583d8d125f724b90608dd2a9d7bbc2defdcaa9198bd3cfaab46b79e6a17" Oct 31 03:26:19.946343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:19.946322 4456 scope.go:122] "RemoveContainer" containerID="c2f8ebd5f6a4ecaa3b7b831d05d195d468dee75fdebde4048c68c81051b2d4ae" Oct 31 03:26:20.434913 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:20.434883 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" path="/var/lib/kubelet/pods/d31b4da9-4b9f-4ff2-85be-343786144bf8/volumes" Oct 31 03:26:20.435391 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:20.435376 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" path="/var/lib/kubelet/pods/da341e75-5d33-4a52-b018-bf1b2d2f83e9/volumes" Oct 31 03:26:26.028099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:26.028055 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f"] Oct 31 03:26:26.032593 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:26.032565 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng"] Oct 31 03:26:26.035700 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:26.035686 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887303fd-s6k6f"] Oct 31 03:26:26.038135 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:26.038116 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971a0b35-xz7ng"] Oct 31 03:26:26.433919 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:26.433885 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4cffa9e-6a33-458c-b16c-203213456d12" path="/var/lib/kubelet/pods/b4cffa9e-6a33-458c-b16c-203213456d12/volumes" Oct 31 03:26:26.434341 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:26:26.434325 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4bc60b4-98d7-4005-97ea-1cadc44f7aab" path="/var/lib/kubelet/pods/e4bc60b4-98d7-4005-97ea-1cadc44f7aab/volumes" Oct 31 03:26:32.133281 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:26:32.133242 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:27:02.132443 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:27:02.132401 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:27:20.035323 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:27:20.035291 4456 scope.go:122] "RemoveContainer" containerID="bd585f24706c9b3a5a3963de4fe87475a29959d870111f3765b4409b5cb58710" Oct 31 03:27:20.045706 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:27:20.045684 4456 scope.go:122] "RemoveContainer" containerID="66edb7929cc5e7894d330735c915030e5e7052947c24f197314553bfd7bc3780" Oct 31 03:27:20.054396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:27:20.054373 4456 scope.go:122] "RemoveContainer" containerID="136b94c3012271e98b5fcf77a7527633a42202c42d4cc049f46cbb10ee4f0b62" Oct 31 03:27:20.063649 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:27:20.063619 4456 scope.go:122] "RemoveContainer" containerID="516846fd86e688b8c3fe6e0120866d949ab9352edfa75cadaaf460aa31ee98f2" Oct 31 03:27:32.133129 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:27:32.133020 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:27:37.029865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:27:37.029831 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb"] Oct 31 03:27:37.032901 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:27:37.032866 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b17858-z4sjb"] Oct 31 03:27:37.035826 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:27:37.035802 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp"] Oct 31 03:27:37.038304 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:27:37.038285 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cf461a-8pwjp"] Oct 31 03:27:38.434149 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:27:38.434108 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b7118d-9b9e-4ab0-856f-f69fe5149e5e" path="/var/lib/kubelet/pods/13b7118d-9b9e-4ab0-856f-f69fe5149e5e/volumes" Oct 31 03:27:38.434781 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:27:38.434713 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe6bbf7-3e18-4ce8-97d7-a8510bcd44df" path="/var/lib/kubelet/pods/abe6bbf7-3e18-4ce8-97d7-a8510bcd44df/volumes" Oct 31 03:28:02.133699 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:28:02.133660 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:28:13.705623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:13.705587 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts"] Oct 31 03:28:13.706177 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:13.705797 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" podUID="cabaf37d-047e-4a35-aa52-646cd02c3496" containerName="discovery" containerID="cri-o://40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8" gracePeriod=30 Oct 31 03:28:13.963081 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:13.962999 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:28:14.064266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.064232 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-token\" (UniqueName: \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-token\") pod \"cabaf37d-047e-4a35-aa52-646cd02c3496\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " Oct 31 03:28:14.064266 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.064270 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-cacerts\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-cacerts\") pod \"cabaf37d-047e-4a35-aa52-646cd02c3496\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " Oct 31 03:28:14.064544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.064294 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-kube-api-access-t9rdn\" (UniqueName: \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-kube-api-access-t9rdn\") pod \"cabaf37d-047e-4a35-aa52-646cd02c3496\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " Oct 31 03:28:14.064544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.064312 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-ca-configmap\") pod \"cabaf37d-047e-4a35-aa52-646cd02c3496\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " Oct 31 03:28:14.064544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.064451 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-kubeconfig\") pod \"cabaf37d-047e-4a35-aa52-646cd02c3496\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " Oct 31 03:28:14.064544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.064491 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabaf37d-047e-4a35-aa52-646cd02c3496-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabaf37d-047e-4a35-aa52-646cd02c3496-local-certs\") pod \"cabaf37d-047e-4a35-aa52-646cd02c3496\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " Oct 31 03:28:14.064781 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.064557 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-dns-cert\") pod \"cabaf37d-047e-4a35-aa52-646cd02c3496\" (UID: \"cabaf37d-047e-4a35-aa52-646cd02c3496\") " Oct 31 03:28:14.064781 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.064731 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-ca-configmap" pod "cabaf37d-047e-4a35-aa52-646cd02c3496" (UID: "cabaf37d-047e-4a35-aa52-646cd02c3496"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:28:14.064925 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.064906 4456 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-ca-configmap\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:14.066647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.066613 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-kube-api-access-t9rdn" pod "cabaf37d-047e-4a35-aa52-646cd02c3496" (UID: "cabaf37d-047e-4a35-aa52-646cd02c3496"). InnerVolumeSpecName "kube-api-access-t9rdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:28:14.066797 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.066775 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-dns-cert" pod "cabaf37d-047e-4a35-aa52-646cd02c3496" (UID: "cabaf37d-047e-4a35-aa52-646cd02c3496"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:28:14.066930 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.066912 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-token" pod "cabaf37d-047e-4a35-aa52-646cd02c3496" (UID: "cabaf37d-047e-4a35-aa52-646cd02c3496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:28:14.067054 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.067020 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-cacerts" pod "cabaf37d-047e-4a35-aa52-646cd02c3496" (UID: "cabaf37d-047e-4a35-aa52-646cd02c3496"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:28:14.067111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.067064 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-kubeconfig" pod "cabaf37d-047e-4a35-aa52-646cd02c3496" (UID: "cabaf37d-047e-4a35-aa52-646cd02c3496"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:28:14.067111 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.067089 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabaf37d-047e-4a35-aa52-646cd02c3496-local-certs" pod "cabaf37d-047e-4a35-aa52-646cd02c3496" (UID: "cabaf37d-047e-4a35-aa52-646cd02c3496"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:28:14.166067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.166003 4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-token\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:14.166067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.166059 4456 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-cacerts\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:14.166307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.166077 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9rdn\" (UniqueName: \"kubernetes.io/projected/cabaf37d-047e-4a35-aa52-646cd02c3496-kube-api-access-t9rdn\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:14.166307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.166094 4456 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-kubeconfig\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:14.166307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.166106 4456 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabaf37d-047e-4a35-aa52-646cd02c3496-local-certs\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:14.166307 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.166117 4456 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cabaf37d-047e-4a35-aa52-646cd02c3496-istio-csr-dns-cert\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:14.686787 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.686738 4456 generic.go:356] "Generic (PLEG): container finished" podID="cabaf37d-047e-4a35-aa52-646cd02c3496" containerID="40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8" exitCode=0 Oct 31 03:28:14.686991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.686799 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" Oct 31 03:28:14.686991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.686810 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" event={"ID":"cabaf37d-047e-4a35-aa52-646cd02c3496","Type":"ContainerDied","Data":"40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8"} Oct 31 03:28:14.686991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.686881 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts" event={"ID":"cabaf37d-047e-4a35-aa52-646cd02c3496","Type":"ContainerDied","Data":"9a32ed62a2b1be9a4968a8b8b097af1add200160601c15437d4cad50d5dc1cb5"} Oct 31 03:28:14.686991 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.686907 4456 scope.go:122] "RemoveContainer" containerID="40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8" Oct 31 03:28:14.701139 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.701105 4456 scope.go:122] "RemoveContainer" containerID="40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8" Oct 31 03:28:14.701393 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:28:14.701371 4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8\": container with ID starting with 40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8 not found: ID does not exist" containerID="40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8" Oct 31 03:28:14.701461 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.701405 4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8"} err="failed to get container status \"40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8\": rpc error: code = NotFound desc = could not find container \"40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8\": container with ID starting with 40f0b885fcfa131ea2184c5667d1ff7256275d224637e1386514fbe1ed6907d8 not found: ID does not exist" Oct 31 03:28:14.717612 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.717560 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts"] Oct 31 03:28:14.722565 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:14.722544 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cnfts"] Oct 31 03:28:16.438574 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:16.438527 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabaf37d-047e-4a35-aa52-646cd02c3496" path="/var/lib/kubelet/pods/cabaf37d-047e-4a35-aa52-646cd02c3496/volumes" Oct 31 03:28:20.105144 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:20.105107 4456 scope.go:122] "RemoveContainer" containerID="1daed2f99f225c05fe0d4f9541d5a2d848a169bf3a7829d16e08ff52187cae10" Oct 31 03:28:20.114835 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:20.114814 4456 scope.go:122] "RemoveContainer" containerID="5525adefd7479e8a5bf04cd9fbbe7c8a38d08a7c20371cb871ba2c844f7c1a22" Oct 31 03:28:20.124109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:20.124090 4456 scope.go:122] "RemoveContainer" containerID="080a7cde0fea8031e8a42cab4b5a57a4069798dcf54bce2f1aa826643f87d4c4" Oct 31 03:28:20.132607 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:20.132589 4456 scope.go:122] "RemoveContainer" containerID="a92d97962f83f9ea1f3e05575eb40c6df637f82a64a5b266bafada66a71e75d3" Oct 31 03:28:20.141334 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:20.141317 4456 scope.go:122] "RemoveContainer" containerID="7239aaf2e9229116dc4f9f227522bfcf96e7c0cea7c7c0b0d3da2cfc97d3b1af" Oct 31 03:28:32.132814 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:28:32.132769 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:28:35.030551 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:35.030517 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb"] Oct 31 03:28:35.033292 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:35.033264 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e726e7-xp9sb"] Oct 31 03:28:36.434746 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:36.434711 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a88f55-d220-468a-8f3e-e1e3c36c628b" path="/var/lib/kubelet/pods/34a88f55-d220-468a-8f3e-e1e3c36c628b/volumes" Oct 31 03:28:42.732280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:42.732236 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6sbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:28:42.732867 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:42.732830 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6sbp" podUID="29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:44.028247 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:44.028210 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf"] Oct 31 03:28:44.031168 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:44.031139 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c4c3764-kkdgf"] Oct 31 03:28:44.434823 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:44.434783 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90285d8-ff96-45d7-949e-5fb48e0ed04a" path="/var/lib/kubelet/pods/f90285d8-ff96-45d7-949e-5fb48e0ed04a/volumes" Oct 31 03:28:45.029208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:45.029175 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj"] Oct 31 03:28:45.031615 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:45.031594 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09924894-xk2rj"] Oct 31 03:28:46.433852 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:46.433815 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af13e2f-3c49-4d1b-baa6-7a9b1959505c" path="/var/lib/kubelet/pods/9af13e2f-3c49-4d1b-baa6-7a9b1959505c/volumes" Oct 31 03:28:47.038108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:47.038078 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78"] Oct 31 03:28:47.040844 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:47.040819 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf5c9d0-7rq78"] Oct 31 03:28:48.434741 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:48.434707 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da158174-c1f6-4b39-a49a-ab75310ab817" path="/var/lib/kubelet/pods/da158174-c1f6-4b39-a49a-ab75310ab817/volumes" Oct 31 03:28:51.067310 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:51.067273 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4"] Oct 31 03:28:51.092358 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:51.092324 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c71d7383-d9lb4"] Oct 31 03:28:52.435068 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:28:52.435001 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966ecee2-b7f6-4bb6-9f05-6af9a19b7274" path="/var/lib/kubelet/pods/966ecee2-b7f6-4bb6-9f05-6af9a19b7274/volumes" Oct 31 03:29:02.132932 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:29:02.132846 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:29:04.034087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:04.034052 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk"] Oct 31 03:29:04.037340 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:04.037321 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e32820f-ddpsk"] Oct 31 03:29:04.437490 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:04.437455 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97db43ad-85cf-42f7-93f2-f83364ecff3d" path="/var/lib/kubelet/pods/97db43ad-85cf-42f7-93f2-f83364ecff3d/volumes" Oct 31 03:29:20.189357 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:20.189325 4456 scope.go:122] "RemoveContainer" containerID="912127e1072d7207aab13870b1b7b0ee766c9aab981b3d9cb77a6626cf5ea9a1" Oct 31 03:29:20.199296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:20.199275 4456 scope.go:122] "RemoveContainer" containerID="1c9004cba6bdc1af0ea8c398e3ead4c6b6e7dee7383ea8b49ba9938b47869cd5" Oct 31 03:29:20.208264 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:20.208237 4456 scope.go:122] "RemoveContainer" containerID="9acad260bb0b2f9e2a0430f0cbc603da54ff1537c6398728b73f52be2fcb7908" Oct 31 03:29:20.217183 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:20.217166 4456 scope.go:122] "RemoveContainer" containerID="5046f0be351fc21e57246b03e70aef8a1b2c257f9c2945ae0cb2e576411b0e89" Oct 31 03:29:32.134930 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:29:32.134872 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:29:55.027499 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:55.027454 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599"] Oct 31 03:29:55.030337 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:55.030313 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-612b4cad-pz599"] Oct 31 03:29:55.032398 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:55.032373 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7"] Oct 31 03:29:55.036108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:55.036086 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a88c35e-26mw7"] Oct 31 03:29:56.435136 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:56.435094 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf8f72a-f23d-47e9-8097-0b50a6217f91" path="/var/lib/kubelet/pods/0cf8f72a-f23d-47e9-8097-0b50a6217f91/volumes" Oct 31 03:29:56.435704 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:29:56.435683 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8a2f48-1a8d-4701-a802-8728187a54dc" path="/var/lib/kubelet/pods/ef8a2f48-1a8d-4701-a802-8728187a54dc/volumes" Oct 31 03:30:02.134132 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:30:02.134092 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:30:06.165866 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.165830 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4"] Oct 31 03:30:06.166342 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166326 4456 cpu_manager.go:414] "containerMap: removing container" podUID="8aae1fe9-9261-421b-b057-80ad4a8fbdef" containerName="revoke-keys" Oct 31 03:30:06.166399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166342 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aae1fe9-9261-421b-b057-80ad4a8fbdef" containerName="revoke-keys" Oct 31 03:30:06.166399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166354 4456 cpu_manager.go:414] "containerMap: removing container" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" Oct 31 03:30:06.166399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166359 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" Oct 31 03:30:06.166399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166369 4456 cpu_manager.go:414] "containerMap: removing container" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" Oct 31 03:30:06.166399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166374 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" Oct 31 03:30:06.166399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166382 4456 cpu_manager.go:414] "containerMap: removing container" podUID="cabaf37d-047e-4a35-aa52-646cd02c3496" containerName="discovery" Oct 31 03:30:06.166399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166387 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabaf37d-047e-4a35-aa52-646cd02c3496" containerName="discovery" Oct 31 03:30:06.166399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166404 4456 cpu_manager.go:414] "containerMap: removing container" podUID="5ecd0946-7571-46bf-9eb7-913be7450dd8" containerName="manager" Oct 31 03:30:06.166730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166409 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ecd0946-7571-46bf-9eb7-913be7450dd8" containerName="manager" Oct 31 03:30:06.166730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166414 4456 cpu_manager.go:414] "containerMap: removing container" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" Oct 31 03:30:06.166730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166419 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" Oct 31 03:30:06.166730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166506 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aae1fe9-9261-421b-b057-80ad4a8fbdef" containerName="revoke-keys" Oct 31 03:30:06.166730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166515 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="da341e75-5d33-4a52-b018-bf1b2d2f83e9" containerName="istio-proxy" Oct 31 03:30:06.166730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166521 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="391ef3a2-cff8-481b-898a-b00a84351d8d" containerName="main" Oct 31 03:30:06.166730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166528 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabaf37d-047e-4a35-aa52-646cd02c3496" containerName="discovery" Oct 31 03:30:06.166730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166538 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ecd0946-7571-46bf-9eb7-913be7450dd8" containerName="manager" Oct 31 03:30:06.166730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.166542 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31b4da9-4b9f-4ff2-85be-343786144bf8" containerName="istio-proxy" Oct 31 03:30:06.170074 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.170056 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.179235 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.179209 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:30:06.179373 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.179278 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:30:06.180117 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.180089 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4"] Oct 31 03:30:06.240996 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.240956 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s4fh4\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.240996 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.240996 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s4fh4\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.241274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.241019 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv58w\" (UniqueName: \"kubernetes.io/projected/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-kube-api-access-nv58w\") pod \"maas-api-revoke-sub-3d4156f1-s4fh4\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.342246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.342207 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s4fh4\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.342246 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.342243 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s4fh4\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.342523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.342264 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv58w\" (UniqueName: \"kubernetes.io/projected/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-kube-api-access-nv58w\") pod \"maas-api-revoke-sub-3d4156f1-s4fh4\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.342962 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.342936 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s4fh4\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.344686 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.344665 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s4fh4\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.355766 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.355741 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv58w\" (UniqueName: \"kubernetes.io/projected/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-kube-api-access-nv58w\") pod \"maas-api-revoke-sub-3d4156f1-s4fh4\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.485651 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.485563 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:06.775303 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:06.774416 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4"] Oct 31 03:30:06.782920 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:30:06.782889 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaca704d_7ae4_40ca_a18c_fa3bbc3d0f72.slice/crio-12c03ce532e3ec8c7e9b9efc16afae9e34a11d09ebf0ded610f051f0552fb974 WatchSource:0}: Error finding container 12c03ce532e3ec8c7e9b9efc16afae9e34a11d09ebf0ded610f051f0552fb974: Status 404 returned error can't find the container with id 12c03ce532e3ec8c7e9b9efc16afae9e34a11d09ebf0ded610f051f0552fb974 Oct 31 03:30:07.546540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:07.546490 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" event={"ID":"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72","Type":"ContainerStarted","Data":"6fdbb2803f6488a59ab8c19fcb87cc21964b86a8862bde6fad7719e24f34d643"} Oct 31 03:30:07.547131 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:07.546552 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" event={"ID":"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72","Type":"ContainerStarted","Data":"12c03ce532e3ec8c7e9b9efc16afae9e34a11d09ebf0ded610f051f0552fb974"} Oct 31 03:30:07.567870 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:07.567817 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" podStartSLOduration=1.567801589 podStartE2EDuration="1.567801589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:07.566640189 +0000 UTC m=+2089.247166317" watchObservedRunningTime="2026-10-31 03:30:07.567801589 +0000 UTC m=+2089.248327733" Oct 31 03:30:08.554999 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:08.554964 4456 generic.go:356] "Generic (PLEG): container finished" podID="aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72" containerID="6fdbb2803f6488a59ab8c19fcb87cc21964b86a8862bde6fad7719e24f34d643" exitCode=0 Oct 31 03:30:08.555423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:08.555074 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" event={"ID":"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72","Type":"ContainerDied","Data":"6fdbb2803f6488a59ab8c19fcb87cc21964b86a8862bde6fad7719e24f34d643"} Oct 31 03:30:09.694209 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:09.694184 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:09.777511 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:09.777475 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-kube-api-access-nv58w\" (UniqueName: \"kubernetes.io/projected/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-kube-api-access-nv58w\") pod \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " Oct 31 03:30:09.777680 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:09.777582 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-openshift-ca\") pod \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " Oct 31 03:30:09.777680 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:09.777623 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-xks-ca\") pod \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\" (UID: \"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72\") " Oct 31 03:30:09.777958 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:09.777935 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-openshift-ca" pod "aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72" (UID: "aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:30:09.779656 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:09.779626 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-xks-ca" pod "aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72" (UID: "aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:30:09.779656 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:09.779635 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-kube-api-access-nv58w" pod "aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72" (UID: "aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72"). InnerVolumeSpecName "kube-api-access-nv58w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:09.878813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:09.878730 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv58w\" (UniqueName: \"kubernetes.io/projected/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-kube-api-access-nv58w\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:09.878813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:09.878762 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:09.878813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:09.878777 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:10.569690 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:10.569661 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" Oct 31 03:30:10.569858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:10.569694 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4" event={"ID":"aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72","Type":"ContainerDied","Data":"12c03ce532e3ec8c7e9b9efc16afae9e34a11d09ebf0ded610f051f0552fb974"} Oct 31 03:30:10.569858 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:10.569741 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12c03ce532e3ec8c7e9b9efc16afae9e34a11d09ebf0ded610f051f0552fb974" Oct 31 03:30:20.256554 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:20.256513 4456 scope.go:122] "RemoveContainer" containerID="7a1725912b303e5e9dfa140d5f1b7b3420f5a1ee42bac0a74c62dae6d71e97b7" Oct 31 03:30:20.276928 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:20.276892 4456 scope.go:122] "RemoveContainer" containerID="d95ae78ceba458726247955a387c745f16e31dce6a03bc26f4f41bd53b5b98c5" Oct 31 03:30:20.288872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:20.288846 4456 scope.go:122] "RemoveContainer" containerID="3f8c5d09b44605606d997ca367b3a33d8145137942cdf33fd76b3dbbb112caf7" Oct 31 03:30:20.300415 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:20.300383 4456 scope.go:122] "RemoveContainer" containerID="bcce3c1f2732183be98d097c0a8c193e05539dc5a55662c3bf9ac0cfb02d6266" Oct 31 03:30:20.311087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:20.311058 4456 scope.go:122] "RemoveContainer" containerID="cdf3c2a0d032e612846a9eb68969a0a952be0c21ff8f0b021c04676a1c4a7b7f" Oct 31 03:30:20.323198 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:20.323175 4456 scope.go:122] "RemoveContainer" containerID="e2142176473fd99d5911132016eed25496a72e70c2e4ac96f66ad7facf12ab1d" Oct 31 03:30:20.336549 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:20.336527 4456 scope.go:122] "RemoveContainer" containerID="2a3104fef36841f2dbf4b328ab88159594679e1196e4b78f93a2a0f3f90a3286" Oct 31 03:30:20.347213 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:20.347159 4456 scope.go:122] "RemoveContainer" containerID="c9bc278c4ab312139c3049218eb97876209f07492b4c022cbd4a1740979e4d8e" Oct 31 03:30:20.359008 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:20.358967 4456 scope.go:122] "RemoveContainer" containerID="8691a0c705ce73674ad3bd7fc19b849992279be0fd997c5a46d2f8d8d221bad4" Oct 31 03:30:20.370604 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:20.370573 4456 scope.go:122] "RemoveContainer" containerID="12b4ff7ff4018f46f7cb6601ce180f082483359c89f4e1768a3a4ce6ddda5cc2" Oct 31 03:30:24.404578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:24.404544 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4"] Oct 31 03:30:24.409140 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:24.409111 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s4fh4"] Oct 31 03:30:24.434359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:24.434324 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72" path="/var/lib/kubelet/pods/aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72/volumes" Oct 31 03:30:32.133756 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:30:32.133673 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:30:54.027376 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:54.027330 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j"] Oct 31 03:30:54.030421 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:54.030395 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed261c50-4h29j"] Oct 31 03:30:54.434108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:30:54.434080 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aae1fe9-9261-421b-b057-80ad4a8fbdef" path="/var/lib/kubelet/pods/8aae1fe9-9261-421b-b057-80ad4a8fbdef/volumes" Oct 31 03:31:02.133764 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:31:02.133721 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:31:20.465434 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:31:20.465404 4456 scope.go:122] "RemoveContainer" containerID="40bfdb356689cc696670d49d4ce9abd19eabe5409c55221d584d4fa9f0941e20" Oct 31 03:31:20.474557 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:31:20.474532 4456 scope.go:122] "RemoveContainer" containerID="aaaad7701568d4daf7595503856446ec609927077172ed1528aa93c6010a82b0" Oct 31 03:31:20.482746 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:31:20.482726 4456 scope.go:122] "RemoveContainer" containerID="82aff105cea612310a59abeca229de21d7820c68639af9a206b2f11e27f2173e" Oct 31 03:31:20.490535 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:31:20.490514 4456 scope.go:122] "RemoveContainer" containerID="11407a46280cac8a683d1cf1df2160396aeaf64f8fbbb57de63e2ebeb8466155" Oct 31 03:31:20.498702 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:31:20.498683 4456 scope.go:122] "RemoveContainer" containerID="265db5e0d8ebbb50c5f7d411ff2f29a15cbc75b8bbbdabadee5282e2ac5dd923" Oct 31 03:31:20.507058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:31:20.507027 4456 scope.go:122] "RemoveContainer" containerID="ea393aaa7584af4609381f65828ca15abad48684cc63193da35c504a38e66c00" Oct 31 03:31:32.144562 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:31:32.144516 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:32:02.133625 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:32:02.133537 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:32:20.552006 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:20.551972 4456 scope.go:122] "RemoveContainer" containerID="e03a98c2413639f65184b9fdd6905a4f61650261fe35958abca84ff285a0781a" Oct 31 03:32:30.435126 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.435098 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb"] Oct 31 03:32:30.435523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.435479 4456 cpu_manager.go:414] "containerMap: removing container" podUID="aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72" containerName="revoke-keys" Oct 31 03:32:30.435523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.435490 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72" containerName="revoke-keys" Oct 31 03:32:30.435598 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.435573 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaca704d-7ae4-40ca-a18c-fa3bbc3d0f72" containerName="revoke-keys" Oct 31 03:32:30.440174 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.440153 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:30.446080 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.446028 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:32:30.448370 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.448342 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:32:30.449408 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.449385 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb"] Oct 31 03:32:30.479812 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.479781 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ed13135-wgxpb\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:30.480010 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.479840 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ed13135-wgxpb\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:30.480109 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.480075 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wv8x\" (UniqueName: \"kubernetes.io/projected/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-kube-api-access-2wv8x\") pod \"maas-api-revoke-sub-0ed13135-wgxpb\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:30.580981 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.580949 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wv8x\" (UniqueName: \"kubernetes.io/projected/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-kube-api-access-2wv8x\") pod \"maas-api-revoke-sub-0ed13135-wgxpb\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:30.581189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.581005 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ed13135-wgxpb\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:30.581189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.581065 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ed13135-wgxpb\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:30.581630 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.581610 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ed13135-wgxpb\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:30.583323 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.583305 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ed13135-wgxpb\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:30.594474 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.594441 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wv8x\" (UniqueName: \"kubernetes.io/projected/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-kube-api-access-2wv8x\") pod \"maas-api-revoke-sub-0ed13135-wgxpb\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:30.754649 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:30.754571 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:31.033201 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:31.033158 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb"] Oct 31 03:32:31.034114 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:32:31.034082 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f3a2e1_066c_47f8_bb0d_cc27bb7399cf.slice/crio-e75f6abd452f06b29c9dc42d8f13193a3952dbd3e6e79f7a02966efd1802885c WatchSource:0}: Error finding container e75f6abd452f06b29c9dc42d8f13193a3952dbd3e6e79f7a02966efd1802885c: Status 404 returned error can't find the container with id e75f6abd452f06b29c9dc42d8f13193a3952dbd3e6e79f7a02966efd1802885c Oct 31 03:32:31.541244 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:31.541194 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" event={"ID":"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf","Type":"ContainerStarted","Data":"fe73c22f6dfd7f615671abd834afc6e95b4fcd7640c00689271766c1142e9561"} Oct 31 03:32:31.541643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:31.541260 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" event={"ID":"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf","Type":"ContainerStarted","Data":"e75f6abd452f06b29c9dc42d8f13193a3952dbd3e6e79f7a02966efd1802885c"} Oct 31 03:32:31.561533 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:31.561481 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" podStartSLOduration=1.5614700049999999 podStartE2EDuration="1.561470005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:31.560968131 +0000 UTC m=+2233.241494258" watchObservedRunningTime="2026-10-31 03:32:31.561470005 +0000 UTC m=+2233.241996132" Oct 31 03:32:32.133454 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:32:32.133368 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:32:32.548495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:32.548457 4456 generic.go:356] "Generic (PLEG): container finished" podID="b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf" containerID="fe73c22f6dfd7f615671abd834afc6e95b4fcd7640c00689271766c1142e9561" exitCode=0 Oct 31 03:32:32.548938 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:32.548547 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" event={"ID":"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf","Type":"ContainerDied","Data":"fe73c22f6dfd7f615671abd834afc6e95b4fcd7640c00689271766c1142e9561"} Oct 31 03:32:33.700115 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:33.700091 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:33.810910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:33.810819 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-openshift-ca\") pod \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " Oct 31 03:32:33.810910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:33.810872 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-xks-ca\") pod \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " Oct 31 03:32:33.810910 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:33.810911 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-kube-api-access-2wv8x\" (UniqueName: \"kubernetes.io/projected/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-kube-api-access-2wv8x\") pod \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\" (UID: \"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf\") " Oct 31 03:32:33.811291 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:33.811268 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-openshift-ca" pod "b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf" (UID: "b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:33.813058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:33.813021 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-kube-api-access-2wv8x" pod "b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf" (UID: "b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf"). InnerVolumeSpecName "kube-api-access-2wv8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:33.813058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:33.813021 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-xks-ca" pod "b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf" (UID: "b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:33.912352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:33.912312 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:33.912352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:33.912345 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:33.912352 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:33.912359 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wv8x\" (UniqueName: \"kubernetes.io/projected/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf-kube-api-access-2wv8x\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:34.561830 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:34.561800 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" Oct 31 03:32:34.562004 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:34.561827 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb" event={"ID":"b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf","Type":"ContainerDied","Data":"e75f6abd452f06b29c9dc42d8f13193a3952dbd3e6e79f7a02966efd1802885c"} Oct 31 03:32:34.562004 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:34.561869 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e75f6abd452f06b29c9dc42d8f13193a3952dbd3e6e79f7a02966efd1802885c" Oct 31 03:32:41.516914 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.516876 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t"] Oct 31 03:32:41.517326 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.517318 4456 cpu_manager.go:414] "containerMap: removing container" podUID="b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf" containerName="revoke-keys" Oct 31 03:32:41.517370 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.517330 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf" containerName="revoke-keys" Oct 31 03:32:41.517421 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.517410 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf" containerName="revoke-keys" Oct 31 03:32:41.520680 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.520665 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:41.526056 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.526014 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:32:41.529484 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.529438 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:32:41.533651 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.533623 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t"] Oct 31 03:32:41.583409 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.583373 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhthg\" (UniqueName: \"kubernetes.io/projected/be0c4106-96c3-4e51-a650-3675cdbe93b0-kube-api-access-vhthg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:41.583600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.583414 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:41.583600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.583441 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:41.685025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.684987 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhthg\" (UniqueName: \"kubernetes.io/projected/be0c4106-96c3-4e51-a650-3675cdbe93b0-kube-api-access-vhthg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:41.685025 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.685025 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:41.685274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.685075 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:41.685728 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.685708 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:41.687537 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.687503 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:41.698508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.698480 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhthg\" (UniqueName: \"kubernetes.io/projected/be0c4106-96c3-4e51-a650-3675cdbe93b0-kube-api-access-vhthg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:41.837343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:41.837257 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:42.118869 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:42.118831 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t"] Oct 31 03:32:42.119826 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:32:42.119798 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0c4106_96c3_4e51_a650_3675cdbe93b0.slice/crio-241d616ce4809541e90efd224e19f27d4ea4c05a1d83d39ff6e4201204da8c38 WatchSource:0}: Error finding container 241d616ce4809541e90efd224e19f27d4ea4c05a1d83d39ff6e4201204da8c38: Status 404 returned error can't find the container with id 241d616ce4809541e90efd224e19f27d4ea4c05a1d83d39ff6e4201204da8c38 Oct 31 03:32:42.611178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:42.611135 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" event={"ID":"be0c4106-96c3-4e51-a650-3675cdbe93b0","Type":"ContainerStarted","Data":"6113eef044af5d42f05bc17a33b91d42d8766638917b8251d7b3c277a22a041a"} Oct 31 03:32:42.611598 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:42.611201 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" event={"ID":"be0c4106-96c3-4e51-a650-3675cdbe93b0","Type":"ContainerStarted","Data":"241d616ce4809541e90efd224e19f27d4ea4c05a1d83d39ff6e4201204da8c38"} Oct 31 03:32:42.633238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:42.633192 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" podStartSLOduration=1.6331781859999999 podStartE2EDuration="1.633178186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:42.63205715 +0000 UTC m=+2244.312583268" watchObservedRunningTime="2026-10-31 03:32:42.633178186 +0000 UTC m=+2244.313704306" Oct 31 03:32:43.619988 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:43.619955 4456 generic.go:356] "Generic (PLEG): container finished" podID="be0c4106-96c3-4e51-a650-3675cdbe93b0" containerID="6113eef044af5d42f05bc17a33b91d42d8766638917b8251d7b3c277a22a041a" exitCode=0 Oct 31 03:32:43.620415 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:43.620008 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" event={"ID":"be0c4106-96c3-4e51-a650-3675cdbe93b0","Type":"ContainerDied","Data":"6113eef044af5d42f05bc17a33b91d42d8766638917b8251d7b3c277a22a041a"} Oct 31 03:32:44.780420 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:44.780392 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:44.917759 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:44.917728 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0c4106-96c3-4e51-a650-3675cdbe93b0-kube-api-access-vhthg\" (UniqueName: \"kubernetes.io/projected/be0c4106-96c3-4e51-a650-3675cdbe93b0-kube-api-access-vhthg\") pod \"be0c4106-96c3-4e51-a650-3675cdbe93b0\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " Oct 31 03:32:44.917939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:44.917797 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-openshift-ca\") pod \"be0c4106-96c3-4e51-a650-3675cdbe93b0\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " Oct 31 03:32:44.917939 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:44.917839 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-xks-ca\") pod \"be0c4106-96c3-4e51-a650-3675cdbe93b0\" (UID: \"be0c4106-96c3-4e51-a650-3675cdbe93b0\") " Oct 31 03:32:44.918271 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:44.918243 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-openshift-ca" pod "be0c4106-96c3-4e51-a650-3675cdbe93b0" (UID: "be0c4106-96c3-4e51-a650-3675cdbe93b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:44.919965 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:44.919936 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-xks-ca" pod "be0c4106-96c3-4e51-a650-3675cdbe93b0" (UID: "be0c4106-96c3-4e51-a650-3675cdbe93b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:44.920105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:44.919997 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0c4106-96c3-4e51-a650-3675cdbe93b0-kube-api-access-vhthg" pod "be0c4106-96c3-4e51-a650-3675cdbe93b0" (UID: "be0c4106-96c3-4e51-a650-3675cdbe93b0"). InnerVolumeSpecName "kube-api-access-vhthg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:45.019647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:45.019608 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhthg\" (UniqueName: \"kubernetes.io/projected/be0c4106-96c3-4e51-a650-3675cdbe93b0-kube-api-access-vhthg\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:45.019647 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:45.019645 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:45.019854 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:45.019659 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0c4106-96c3-4e51-a650-3675cdbe93b0-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:45.632907 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:45.632858 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" event={"ID":"be0c4106-96c3-4e51-a650-3675cdbe93b0","Type":"ContainerDied","Data":"241d616ce4809541e90efd224e19f27d4ea4c05a1d83d39ff6e4201204da8c38"} Oct 31 03:32:45.633107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:45.632929 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="241d616ce4809541e90efd224e19f27d4ea4c05a1d83d39ff6e4201204da8c38" Oct 31 03:32:45.633107 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:45.632872 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t" Oct 31 03:32:52.710767 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:52.710728 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t"] Oct 31 03:32:52.713194 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:52.713172 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-01b345-p6h9t"] Oct 31 03:32:54.434055 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:32:54.434006 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be0c4106-96c3-4e51-a650-3675cdbe93b0" path="/var/lib/kubelet/pods/be0c4106-96c3-4e51-a650-3675cdbe93b0/volumes" Oct 31 03:33:02.133502 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:33:02.133464 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:33:20.315801 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.315763 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc"] Oct 31 03:33:20.316332 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.316010 4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" podUID="92f6df64-c757-4734-9cb0-59d3558b538b" containerName="manager" containerID="cri-o://a095de909e2b640036ec8b4b4ff44f580dcbd1bda514598426d2f9e0177f6a66" gracePeriod=10 Oct 31 03:33:20.573763 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.573698 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:33:20.575626 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.575605 4456 scope.go:122] "RemoveContainer" containerID="a095de909e2b640036ec8b4b4ff44f580dcbd1bda514598426d2f9e0177f6a66" Oct 31 03:33:20.580628 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.580608 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f6df64-c757-4734-9cb0-59d3558b538b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92f6df64-c757-4734-9cb0-59d3558b538b-extensions-socket-volume\") pod \"92f6df64-c757-4734-9cb0-59d3558b538b\" (UID: \"92f6df64-c757-4734-9cb0-59d3558b538b\") " Oct 31 03:33:20.580735 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.580642 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f6df64-c757-4734-9cb0-59d3558b538b-kube-api-access-dp4hf\" (UniqueName: \"kubernetes.io/projected/92f6df64-c757-4734-9cb0-59d3558b538b-kube-api-access-dp4hf\") pod \"92f6df64-c757-4734-9cb0-59d3558b538b\" (UID: \"92f6df64-c757-4734-9cb0-59d3558b538b\") " Oct 31 03:33:20.581024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.581002 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f6df64-c757-4734-9cb0-59d3558b538b-extensions-socket-volume" pod "92f6df64-c757-4734-9cb0-59d3558b538b" (UID: "92f6df64-c757-4734-9cb0-59d3558b538b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:20.582623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.582603 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f6df64-c757-4734-9cb0-59d3558b538b-kube-api-access-dp4hf" pod "92f6df64-c757-4734-9cb0-59d3558b538b" (UID: "92f6df64-c757-4734-9cb0-59d3558b538b"). InnerVolumeSpecName "kube-api-access-dp4hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:20.681228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.681190 4456 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92f6df64-c757-4734-9cb0-59d3558b538b-extensions-socket-volume\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:20.681228 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.681217 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4hf\" (UniqueName: \"kubernetes.io/projected/92f6df64-c757-4734-9cb0-59d3558b538b-kube-api-access-dp4hf\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:20.855794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.855694 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" event={"ID":"92f6df64-c757-4734-9cb0-59d3558b538b","Type":"ContainerDied","Data":"a095de909e2b640036ec8b4b4ff44f580dcbd1bda514598426d2f9e0177f6a66"} Oct 31 03:33:20.855794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.855765 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" event={"ID":"92f6df64-c757-4734-9cb0-59d3558b538b","Type":"ContainerDied","Data":"95c252cb35f3603347f261eb9a90b52a7cce597d8c7deaa504fe74b20c58ea4c"} Oct 31 03:33:20.855794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.855712 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc" Oct 31 03:33:20.896699 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.896657 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc"] Oct 31 03:33:20.900370 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:20.900343 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xgwpc"] Oct 31 03:33:22.434400 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:33:22.434363 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f6df64-c757-4734-9cb0-59d3558b538b" path="/var/lib/kubelet/pods/92f6df64-c757-4734-9cb0-59d3558b538b/volumes" Oct 31 03:33:32.133645 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:33:32.133566 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:34:02.132185 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:34:02.132144 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:34:21.360883 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.360845 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm"] Oct 31 03:34:21.361363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.361278 4456 cpu_manager.go:414] "containerMap: removing container" podUID="be0c4106-96c3-4e51-a650-3675cdbe93b0" containerName="revoke-keys" Oct 31 03:34:21.361363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.361290 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0c4106-96c3-4e51-a650-3675cdbe93b0" containerName="revoke-keys" Oct 31 03:34:21.361363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.361320 4456 cpu_manager.go:414] "containerMap: removing container" podUID="92f6df64-c757-4734-9cb0-59d3558b538b" containerName="manager" Oct 31 03:34:21.361363 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.361325 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f6df64-c757-4734-9cb0-59d3558b538b" containerName="manager" Oct 31 03:34:21.361499 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.361392 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f6df64-c757-4734-9cb0-59d3558b538b" containerName="manager" Oct 31 03:34:21.361499 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.361405 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0c4106-96c3-4e51-a650-3675cdbe93b0" containerName="revoke-keys" Oct 31 03:34:21.364522 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.364503 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" Oct 31 03:34:21.371463 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.371321 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-swfhz\"" Oct 31 03:34:21.377354 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.377329 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm"] Oct 31 03:34:21.458141 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.458108 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf488\" (UniqueName: \"kubernetes.io/projected/4e1ff94e-bee2-4e63-beda-55fed25dbfbe-kube-api-access-zf488\") pod \"kuadrant-operator-controller-manager-678d87766f-gjpzm\" (UID: \"4e1ff94e-bee2-4e63-beda-55fed25dbfbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" Oct 31 03:34:21.458318 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.458164 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e1ff94e-bee2-4e63-beda-55fed25dbfbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gjpzm\" (UID: \"4e1ff94e-bee2-4e63-beda-55fed25dbfbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" Oct 31 03:34:21.559726 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.559694 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf488\" (UniqueName: \"kubernetes.io/projected/4e1ff94e-bee2-4e63-beda-55fed25dbfbe-kube-api-access-zf488\") pod \"kuadrant-operator-controller-manager-678d87766f-gjpzm\" (UID: \"4e1ff94e-bee2-4e63-beda-55fed25dbfbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" Oct 31 03:34:21.559926 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.559743 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e1ff94e-bee2-4e63-beda-55fed25dbfbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gjpzm\" (UID: \"4e1ff94e-bee2-4e63-beda-55fed25dbfbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" Oct 31 03:34:21.560180 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.560162 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e1ff94e-bee2-4e63-beda-55fed25dbfbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gjpzm\" (UID: \"4e1ff94e-bee2-4e63-beda-55fed25dbfbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" Oct 31 03:34:21.573892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.573860 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf488\" (UniqueName: \"kubernetes.io/projected/4e1ff94e-bee2-4e63-beda-55fed25dbfbe-kube-api-access-zf488\") pod \"kuadrant-operator-controller-manager-678d87766f-gjpzm\" (UID: \"4e1ff94e-bee2-4e63-beda-55fed25dbfbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" Oct 31 03:34:21.678069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.678013 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" Oct 31 03:34:21.957896 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:21.957862 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm"] Oct 31 03:34:21.958993 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:34:21.958952 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1ff94e_bee2_4e63_beda_55fed25dbfbe.slice/crio-25c8100ef56a2eec2b9b90e9f2f47f65c9fea640bd1bfb32be8fca6b63051c66 WatchSource:0}: Error finding container 25c8100ef56a2eec2b9b90e9f2f47f65c9fea640bd1bfb32be8fca6b63051c66: Status 404 returned error can't find the container with id 25c8100ef56a2eec2b9b90e9f2f47f65c9fea640bd1bfb32be8fca6b63051c66 Oct 31 03:34:22.201276 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:22.201244 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" event={"ID":"4e1ff94e-bee2-4e63-beda-55fed25dbfbe","Type":"ContainerStarted","Data":"95f7b51d097dd0b8228d6f857487064d848c0751cc5a25cc9709a895c24ebcfa"} Oct 31 03:34:22.201452 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:22.201292 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" event={"ID":"4e1ff94e-bee2-4e63-beda-55fed25dbfbe","Type":"ContainerStarted","Data":"25c8100ef56a2eec2b9b90e9f2f47f65c9fea640bd1bfb32be8fca6b63051c66"} Oct 31 03:34:22.201452 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:22.201367 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" Oct 31 03:34:22.225772 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:22.225689 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" podStartSLOduration=1.225677988 podStartE2EDuration="1.225677988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:22.225316228 +0000 UTC m=+2343.905842355" watchObservedRunningTime="2026-10-31 03:34:22.225677988 +0000 UTC m=+2343.906204115" Oct 31 03:34:32.133061 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:34:32.133002 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:34:33.209733 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:34:33.209703 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gjpzm" Oct 31 03:35:02.133314 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:35:02.133232 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:35:32.133797 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:35:32.133762 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:36:02.137165 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:36:02.137118 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:36:08.573705 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.573663 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml"] Oct 31 03:36:08.577540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.577516 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:08.586600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.585685 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:36:08.586600 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.585946 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:36:08.588531 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.588506 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml"] Oct 31 03:36:08.657139 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.657106 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbzp\" (UniqueName: \"kubernetes.io/projected/d021a2f4-b742-4017-9b00-3f9ca6108bd8-kube-api-access-zsbzp\") pod \"maas-api-revoke-sub-6efe1033-8grml\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:08.657345 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.657168 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6efe1033-8grml\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:08.657345 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.657326 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6efe1033-8grml\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:08.758395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.758358 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6efe1033-8grml\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:08.758587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.758449 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbzp\" (UniqueName: \"kubernetes.io/projected/d021a2f4-b742-4017-9b00-3f9ca6108bd8-kube-api-access-zsbzp\") pod \"maas-api-revoke-sub-6efe1033-8grml\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:08.758587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.758486 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6efe1033-8grml\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:08.759225 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.759201 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6efe1033-8grml\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:08.760780 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.760758 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6efe1033-8grml\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:08.772555 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.772532 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbzp\" (UniqueName: \"kubernetes.io/projected/d021a2f4-b742-4017-9b00-3f9ca6108bd8-kube-api-access-zsbzp\") pod \"maas-api-revoke-sub-6efe1033-8grml\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:08.894381 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:08.894297 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:09.177874 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:09.177837 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml"] Oct 31 03:36:09.182962 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:36:09.182926 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd021a2f4_b742_4017_9b00_3f9ca6108bd8.slice/crio-ef719cf4fa7b4c85f8e0efa51949098c80b1bb3bb691586364799d425494a8fb WatchSource:0}: Error finding container ef719cf4fa7b4c85f8e0efa51949098c80b1bb3bb691586364799d425494a8fb: Status 404 returned error can't find the container with id ef719cf4fa7b4c85f8e0efa51949098c80b1bb3bb691586364799d425494a8fb Oct 31 03:36:09.854613 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:09.854524 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" event={"ID":"d021a2f4-b742-4017-9b00-3f9ca6108bd8","Type":"ContainerStarted","Data":"d437f6e96c2326bd6c54ca486abb98eceb26c55c7900328ae891b22fdb2bf392"} Oct 31 03:36:09.854613 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:09.854584 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" event={"ID":"d021a2f4-b742-4017-9b00-3f9ca6108bd8","Type":"ContainerStarted","Data":"ef719cf4fa7b4c85f8e0efa51949098c80b1bb3bb691586364799d425494a8fb"} Oct 31 03:36:09.876711 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:09.876662 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" podStartSLOduration=1.876649005 podStartE2EDuration="1.876649005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:09.875528071 +0000 UTC m=+2451.556054201" watchObservedRunningTime="2026-10-31 03:36:09.876649005 +0000 UTC m=+2451.557175131" Oct 31 03:36:10.870102 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:10.870063 4456 generic.go:356] "Generic (PLEG): container finished" podID="d021a2f4-b742-4017-9b00-3f9ca6108bd8" containerID="d437f6e96c2326bd6c54ca486abb98eceb26c55c7900328ae891b22fdb2bf392" exitCode=0 Oct 31 03:36:10.870495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:10.870076 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" event={"ID":"d021a2f4-b742-4017-9b00-3f9ca6108bd8","Type":"ContainerDied","Data":"d437f6e96c2326bd6c54ca486abb98eceb26c55c7900328ae891b22fdb2bf392"} Oct 31 03:36:12.011724 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.011700 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:12.095046 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.095012 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d021a2f4-b742-4017-9b00-3f9ca6108bd8-kube-api-access-zsbzp\" (UniqueName: \"kubernetes.io/projected/d021a2f4-b742-4017-9b00-3f9ca6108bd8-kube-api-access-zsbzp\") pod \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " Oct 31 03:36:12.095260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.095163 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-openshift-ca\") pod \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " Oct 31 03:36:12.095260 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.095198 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-xks-ca\") pod \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\" (UID: \"d021a2f4-b742-4017-9b00-3f9ca6108bd8\") " Oct 31 03:36:12.095538 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.095511 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-openshift-ca" pod "d021a2f4-b742-4017-9b00-3f9ca6108bd8" (UID: "d021a2f4-b742-4017-9b00-3f9ca6108bd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:12.097251 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.097222 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-xks-ca" pod "d021a2f4-b742-4017-9b00-3f9ca6108bd8" (UID: "d021a2f4-b742-4017-9b00-3f9ca6108bd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:36:12.097379 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.097355 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d021a2f4-b742-4017-9b00-3f9ca6108bd8-kube-api-access-zsbzp" pod "d021a2f4-b742-4017-9b00-3f9ca6108bd8" (UID: "d021a2f4-b742-4017-9b00-3f9ca6108bd8"). InnerVolumeSpecName "kube-api-access-zsbzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:12.196863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.196829 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsbzp\" (UniqueName: \"kubernetes.io/projected/d021a2f4-b742-4017-9b00-3f9ca6108bd8-kube-api-access-zsbzp\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:12.196863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.196860 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:12.196863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.196871 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d021a2f4-b742-4017-9b00-3f9ca6108bd8-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:12.884311 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.884281 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" Oct 31 03:36:12.884311 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.884297 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml" event={"ID":"d021a2f4-b742-4017-9b00-3f9ca6108bd8","Type":"ContainerDied","Data":"ef719cf4fa7b4c85f8e0efa51949098c80b1bb3bb691586364799d425494a8fb"} Oct 31 03:36:12.884540 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:12.884340 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef719cf4fa7b4c85f8e0efa51949098c80b1bb3bb691586364799d425494a8fb" Oct 31 03:36:19.500426 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.500391 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt"] Oct 31 03:36:19.500934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.500872 4456 cpu_manager.go:414] "containerMap: removing container" podUID="d021a2f4-b742-4017-9b00-3f9ca6108bd8" containerName="revoke-keys" Oct 31 03:36:19.500934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.500885 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d021a2f4-b742-4017-9b00-3f9ca6108bd8" containerName="revoke-keys" Oct 31 03:36:19.501130 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.500976 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d021a2f4-b742-4017-9b00-3f9ca6108bd8" containerName="revoke-keys" Oct 31 03:36:19.504290 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.504268 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:19.510312 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.510284 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:36:19.514147 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.514119 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt"] Oct 31 03:36:19.515516 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.515491 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:36:19.671593 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.671558 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6j8\" (UniqueName: \"kubernetes.io/projected/573d529b-b359-45f9-a40b-900cf3f0db89-kube-api-access-6m6j8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:19.671593 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.671595 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:19.671817 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.671708 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:19.773305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.773191 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6j8\" (UniqueName: \"kubernetes.io/projected/573d529b-b359-45f9-a40b-900cf3f0db89-kube-api-access-6m6j8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:19.773305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.773257 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:19.773587 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.773397 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:19.774069 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.774025 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:19.775913 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.775883 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:19.790613 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.790593 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6j8\" (UniqueName: \"kubernetes.io/projected/573d529b-b359-45f9-a40b-900cf3f0db89-kube-api-access-6m6j8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:19.822776 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:19.822743 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:20.110671 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:20.110634 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt"] Oct 31 03:36:20.112050 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:36:20.111999 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573d529b_b359_45f9_a40b_900cf3f0db89.slice/crio-103c1a4fdd96361ec7b4579bd03c2dcd9d9465ead8b620d2416f23684db33b29 WatchSource:0}: Error finding container 103c1a4fdd96361ec7b4579bd03c2dcd9d9465ead8b620d2416f23684db33b29: Status 404 returned error can't find the container with id 103c1a4fdd96361ec7b4579bd03c2dcd9d9465ead8b620d2416f23684db33b29 Oct 31 03:36:20.635606 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:20.635528 4456 scope.go:122] "RemoveContainer" containerID="6fdbb2803f6488a59ab8c19fcb87cc21964b86a8862bde6fad7719e24f34d643" Oct 31 03:36:20.941645 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:20.941608 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" event={"ID":"573d529b-b359-45f9-a40b-900cf3f0db89","Type":"ContainerStarted","Data":"6a001ad9e1c3867d573e89dc2fb61f25f460ea41820f85f83f48e3dd5067ea80"} Oct 31 03:36:20.941857 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:20.941666 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" event={"ID":"573d529b-b359-45f9-a40b-900cf3f0db89","Type":"ContainerStarted","Data":"103c1a4fdd96361ec7b4579bd03c2dcd9d9465ead8b620d2416f23684db33b29"} Oct 31 03:36:20.964664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:20.964618 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" podStartSLOduration=1.964605534 podStartE2EDuration="1.964605534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:20.962386172 +0000 UTC m=+2462.642912312" watchObservedRunningTime="2026-10-31 03:36:20.964605534 +0000 UTC m=+2462.645131694" Oct 31 03:36:21.948508 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:21.948472 4456 generic.go:356] "Generic (PLEG): container finished" podID="573d529b-b359-45f9-a40b-900cf3f0db89" containerID="6a001ad9e1c3867d573e89dc2fb61f25f460ea41820f85f83f48e3dd5067ea80" exitCode=0 Oct 31 03:36:21.948954 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:21.948563 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" event={"ID":"573d529b-b359-45f9-a40b-900cf3f0db89","Type":"ContainerDied","Data":"6a001ad9e1c3867d573e89dc2fb61f25f460ea41820f85f83f48e3dd5067ea80"} Oct 31 03:36:23.092860 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.092835 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:23.204358 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.204324 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-openshift-ca\") pod \"573d529b-b359-45f9-a40b-900cf3f0db89\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " Oct 31 03:36:23.204570 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.204384 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573d529b-b359-45f9-a40b-900cf3f0db89-kube-api-access-6m6j8\" (UniqueName: \"kubernetes.io/projected/573d529b-b359-45f9-a40b-900cf3f0db89-kube-api-access-6m6j8\") pod \"573d529b-b359-45f9-a40b-900cf3f0db89\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " Oct 31 03:36:23.204570 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.204405 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-xks-ca\") pod \"573d529b-b359-45f9-a40b-900cf3f0db89\" (UID: \"573d529b-b359-45f9-a40b-900cf3f0db89\") " Oct 31 03:36:23.204766 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.204742 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-openshift-ca" pod "573d529b-b359-45f9-a40b-900cf3f0db89" (UID: "573d529b-b359-45f9-a40b-900cf3f0db89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:23.206424 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.206362 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573d529b-b359-45f9-a40b-900cf3f0db89-kube-api-access-6m6j8" pod "573d529b-b359-45f9-a40b-900cf3f0db89" (UID: "573d529b-b359-45f9-a40b-900cf3f0db89"). InnerVolumeSpecName "kube-api-access-6m6j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:23.206424 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.206377 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-xks-ca" pod "573d529b-b359-45f9-a40b-900cf3f0db89" (UID: "573d529b-b359-45f9-a40b-900cf3f0db89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:36:23.305882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.305845 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:23.305882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.305876 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6j8\" (UniqueName: \"kubernetes.io/projected/573d529b-b359-45f9-a40b-900cf3f0db89-kube-api-access-6m6j8\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:23.305882 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.305887 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573d529b-b359-45f9-a40b-900cf3f0db89-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:23.963095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.963062 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" Oct 31 03:36:23.963294 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.963066 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt" event={"ID":"573d529b-b359-45f9-a40b-900cf3f0db89","Type":"ContainerDied","Data":"103c1a4fdd96361ec7b4579bd03c2dcd9d9465ead8b620d2416f23684db33b29"} Oct 31 03:36:23.963294 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:23.963201 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="103c1a4fdd96361ec7b4579bd03c2dcd9d9465ead8b620d2416f23684db33b29" Oct 31 03:36:30.859384 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:30.859295 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt"] Oct 31 03:36:30.861934 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:30.861907 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-31e475-vg8lt"] Oct 31 03:36:32.133409 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:36:32.133374 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:36:32.435202 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:36:32.435172 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="573d529b-b359-45f9-a40b-900cf3f0db89" path="/var/lib/kubelet/pods/573d529b-b359-45f9-a40b-900cf3f0db89/volumes" Oct 31 03:37:02.186445 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:37:02.186346 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:37:20.562060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:37:20.561140 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7r8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:37:20.562060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:37:20.561233 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" podUID="ef33ec8e-50ff-4f0b-a84c-25a9b07c2221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:37:32.220534 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:37:32.220468 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:37:34.028149 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:37:34.028087 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb"] Oct 31 03:37:34.031786 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:37:34.031760 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ed13135-wgxpb"] Oct 31 03:37:34.437165 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:37:34.437126 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf" path="/var/lib/kubelet/pods/b9f3a2e1-066c-47f8-bb0d-cc27bb7399cf/volumes" Oct 31 03:37:35.562263 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:37:35.562206 4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7r8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:37:35.562948 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:37:35.562915 4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7r8z" podUID="ef33ec8e-50ff-4f0b-a84c-25a9b07c2221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:02.134116 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:38:02.134008 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:38:27.756751 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.756714 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c"] Oct 31 03:38:27.757197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.757153 4456 cpu_manager.go:414] "containerMap: removing container" podUID="573d529b-b359-45f9-a40b-900cf3f0db89" containerName="revoke-keys" Oct 31 03:38:27.757197 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.757166 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573d529b-b359-45f9-a40b-900cf3f0db89" containerName="revoke-keys" Oct 31 03:38:27.757280 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.757261 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="573d529b-b359-45f9-a40b-900cf3f0db89" containerName="revoke-keys" Oct 31 03:38:27.760461 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.760445 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:27.765815 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.765783 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:38:27.769082 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.769026 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:38:27.769189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.769088 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c"] Oct 31 03:38:27.842840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.842805 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7msg\" (UniqueName: \"kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:27.842840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.842844 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:27.843074 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.842874 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:27.943994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.943960 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7msg\" (UniqueName: \"kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:27.943994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.943996 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:27.944251 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.944032 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:27.944644 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.944619 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:27.946447 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:27.946418 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:27.955732 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:38:27.955715 4456 projected.go:196] Error preparing data for projected volume kube-api-access-w7msg for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 03:38:27.955822 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:38:27.955785 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg podName:cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0 nodeName:}" failed. No retries permitted until 2026-10-31 03:38:28.455771591 +0000 UTC m=+2590.136297714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w7msg" (UniqueName: "kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg") pod "maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" (UID: "cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 03:38:28.548712 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:28.548671 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7msg\" (UniqueName: \"kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:28.553558 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:38:28.553534 4456 projected.go:196] Error preparing data for projected volume kube-api-access-w7msg for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 03:38:28.553653 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:38:28.553603 4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg podName:cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0 nodeName:}" failed. No retries permitted until 2026-10-31 03:38:29.553584752 +0000 UTC m=+2591.234110856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7msg" (UniqueName: "kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg") pod "maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" (UID: "cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 03:38:29.560454 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:29.560417 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7msg\" (UniqueName: \"kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:29.573967 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:29.573934 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7msg\" (UniqueName: \"kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg\") pod \"maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:29.575573 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:29.575551 4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" secret="" err="secret \"maas-api-cleanup-dockercfg-54pxf\" not found" Oct 31 03:38:29.575639 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:29.575598 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:29.855915 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:29.855887 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c"] Oct 31 03:38:29.856513 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:38:29.856483 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1720a7_54ea_4cd2_af35_5d3d5ebc0cc0.slice/crio-62e6aaf0c43261b54f33d1cb61fb8c0af5a4dc0dfe489696b37f8a03d2fa6baf WatchSource:0}: Error finding container 62e6aaf0c43261b54f33d1cb61fb8c0af5a4dc0dfe489696b37f8a03d2fa6baf: Status 404 returned error can't find the container with id 62e6aaf0c43261b54f33d1cb61fb8c0af5a4dc0dfe489696b37f8a03d2fa6baf Oct 31 03:38:29.893811 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:29.893788 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" event={"ID":"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0","Type":"ContainerStarted","Data":"62e6aaf0c43261b54f33d1cb61fb8c0af5a4dc0dfe489696b37f8a03d2fa6baf"} Oct 31 03:38:30.902208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:30.902170 4456 generic.go:356] "Generic (PLEG): container finished" podID="cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0" containerID="4582ae5facfe74833f0bea1df9670a0c93f29813f150be9266b76343c0385414" exitCode=0 Oct 31 03:38:30.902623 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:30.902248 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" event={"ID":"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0","Type":"ContainerDied","Data":"4582ae5facfe74833f0bea1df9670a0c93f29813f150be9266b76343c0385414"} Oct 31 03:38:32.044679 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.044643 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:32.084248 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.084218 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-xks-ca\") pod \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " Oct 31 03:38:32.084402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.084258 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-openshift-ca\") pod \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " Oct 31 03:38:32.084402 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.084290 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg\" (UniqueName: \"kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg\") pod \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\" (UID: \"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0\") " Oct 31 03:38:32.084682 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.084643 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-openshift-ca" pod "cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0" (UID: "cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:32.086481 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.086456 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-xks-ca" pod "cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0" (UID: "cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:32.086588 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.086547 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg" pod "cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0" (UID: "cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0"). InnerVolumeSpecName "kube-api-access-w7msg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:32.134391 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:38:32.134353 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:38:32.185701 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.185628 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:32.185701 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.185654 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:32.185701 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.185665 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7msg\" (UniqueName: \"kubernetes.io/projected/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0-kube-api-access-w7msg\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:32.914863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.914836 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" Oct 31 03:38:32.915078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.914871 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c" event={"ID":"cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0","Type":"ContainerDied","Data":"62e6aaf0c43261b54f33d1cb61fb8c0af5a4dc0dfe489696b37f8a03d2fa6baf"} Oct 31 03:38:32.915078 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:32.914918 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62e6aaf0c43261b54f33d1cb61fb8c0af5a4dc0dfe489696b37f8a03d2fa6baf" Oct 31 03:38:38.551959 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:38.551914 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c"] Oct 31 03:38:38.555929 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:38.555899 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17e11a7f-c9w4c"] Oct 31 03:38:40.433822 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:38:40.433781 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0" path="/var/lib/kubelet/pods/cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0/volumes" Oct 31 03:39:02.133508 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:39:02.133387 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:39:20.722885 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:39:20.722849 4456 scope.go:122] "RemoveContainer" containerID="6113eef044af5d42f05bc17a33b91d42d8766638917b8251d7b3c277a22a041a" Oct 31 03:39:20.731964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:39:20.731942 4456 scope.go:122] "RemoveContainer" containerID="fe73c22f6dfd7f615671abd834afc6e95b4fcd7640c00689271766c1142e9561" Oct 31 03:39:32.133822 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:39:32.133734 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:40:02.133125 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:40:02.133084 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:40:24.993648 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:24.993613 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt"] Oct 31 03:40:24.994108 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:24.994088 4456 cpu_manager.go:414] "containerMap: removing container" podUID="cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0" containerName="revoke-keys" Oct 31 03:40:24.994161 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:24.994110 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0" containerName="revoke-keys" Oct 31 03:40:24.994208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:24.994198 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1720a7-54ea-4cd2-af35-5d3d5ebc0cc0" containerName="revoke-keys" Oct 31 03:40:24.997409 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:24.997394 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.002690 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.002666 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:40:25.004621 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.004600 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kljfr\"" Oct 31 03:40:25.009624 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.009603 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt"] Oct 31 03:40:25.061031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.061003 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v5qc\" (UniqueName: \"kubernetes.io/projected/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-kube-api-access-4v5qc\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.061229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.061056 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.061306 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.061224 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.162471 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.162442 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v5qc\" (UniqueName: \"kubernetes.io/projected/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-kube-api-access-4v5qc\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.162609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.162481 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.162609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.162541 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.163237 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.163211 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.164767 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.164749 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.175785 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.175765 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v5qc\" (UniqueName: \"kubernetes.io/projected/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-kube-api-access-4v5qc\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.313502 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.313408 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:25.591215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:25.591185 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt"] Oct 31 03:40:25.591961 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:40:25.591923 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc21b7ec_6460_4cb8_9aa5_c0203abd1ccb.slice/crio-c564a2ed75aef3b5a2c656af014b1863746d8e0c420fbcde657e098523eadc30 WatchSource:0}: Error finding container c564a2ed75aef3b5a2c656af014b1863746d8e0c420fbcde657e098523eadc30: Status 404 returned error can't find the container with id c564a2ed75aef3b5a2c656af014b1863746d8e0c420fbcde657e098523eadc30 Oct 31 03:40:26.580889 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:26.580851 4456 generic.go:356] "Generic (PLEG): container finished" podID="dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb" containerID="418db2f25fe5bf41ecb316b473ce06c3980dc63a32449c671f37899ec3fe90da" exitCode=0 Oct 31 03:40:26.581318 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:26.580939 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" event={"ID":"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb","Type":"ContainerDied","Data":"418db2f25fe5bf41ecb316b473ce06c3980dc63a32449c671f37899ec3fe90da"} Oct 31 03:40:26.581318 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:26.580994 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" event={"ID":"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb","Type":"ContainerStarted","Data":"c564a2ed75aef3b5a2c656af014b1863746d8e0c420fbcde657e098523eadc30"} Oct 31 03:40:27.721437 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:27.721408 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:27.786778 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:27.786734 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-openshift-ca\") pod \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " Oct 31 03:40:27.786960 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:27.786828 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-kube-api-access-4v5qc\" (UniqueName: \"kubernetes.io/projected/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-kube-api-access-4v5qc\") pod \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " Oct 31 03:40:27.786960 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:27.786922 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-xks-ca\") pod \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\" (UID: \"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb\") " Oct 31 03:40:27.787210 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:27.787185 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-openshift-ca" pod "dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb" (UID: "dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:27.788917 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:27.788893 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-xks-ca" pod "dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb" (UID: "dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:27.789032 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:27.788917 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-kube-api-access-4v5qc" pod "dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb" (UID: "dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb"). InnerVolumeSpecName "kube-api-access-4v5qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:27.888156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:27.888073 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:27.888156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:27.888102 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v5qc\" (UniqueName: \"kubernetes.io/projected/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-kube-api-access-4v5qc\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:27.888156 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:27.888111 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:28.594288 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:28.594257 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" Oct 31 03:40:28.594510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:28.594297 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt" event={"ID":"dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb","Type":"ContainerDied","Data":"c564a2ed75aef3b5a2c656af014b1863746d8e0c420fbcde657e098523eadc30"} Oct 31 03:40:28.594510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:28.594342 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c564a2ed75aef3b5a2c656af014b1863746d8e0c420fbcde657e098523eadc30" Oct 31 03:40:32.133552 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:40:32.133513 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:40:41.155702 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:41.155667 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt"] Oct 31 03:40:41.158174 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:41.158151 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d20d2-g5wkt"] Oct 31 03:40:42.437477 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:40:42.437442 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb" path="/var/lib/kubelet/pods/dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb/volumes" Oct 31 03:41:00.187961 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.187917 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j"] Oct 31 03:41:00.188399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.188358 4456 cpu_manager.go:414] "containerMap: removing container" podUID="dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb" containerName="revoke-keys" Oct 31 03:41:00.188399 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.188370 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb" containerName="revoke-keys" Oct 31 03:41:00.188486 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.188453 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc21b7ec-6460-4cb8-9aa5-c0203abd1ccb" containerName="revoke-keys" Oct 31 03:41:00.191803 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.191785 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.196988 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.196963 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:41:00.198917 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.198898 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kljfr\"" Oct 31 03:41:00.204523 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.204498 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j"] Oct 31 03:41:00.342164 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.342130 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rlq5j\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.342389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.342173 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rlq5j\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.342389 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.342208 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghts\" (UniqueName: \"kubernetes.io/projected/f7f96aea-6782-4e48-ac8a-019f67f2352c-kube-api-access-nghts\") pod \"maas-api-revoke-sub-f03629db-rlq5j\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.443424 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.443342 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rlq5j\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.443424 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.443386 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rlq5j\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.443669 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.443429 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghts\" (UniqueName: \"kubernetes.io/projected/f7f96aea-6782-4e48-ac8a-019f67f2352c-kube-api-access-nghts\") pod \"maas-api-revoke-sub-f03629db-rlq5j\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.444118 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.444098 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rlq5j\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.445725 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.445707 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rlq5j\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.456964 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.456942 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghts\" (UniqueName: \"kubernetes.io/projected/f7f96aea-6782-4e48-ac8a-019f67f2352c-kube-api-access-nghts\") pod \"maas-api-revoke-sub-f03629db-rlq5j\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.506622 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.506596 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:00.785655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.785622 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j"] Oct 31 03:41:00.785813 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:41:00.785757 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f96aea_6782_4e48_ac8a_019f67f2352c.slice/crio-4a2e5b5679eb4518c591962eb63eefd5862e3b6fe68537ba85deba2e89e21bb0 WatchSource:0}: Error finding container 4a2e5b5679eb4518c591962eb63eefd5862e3b6fe68537ba85deba2e89e21bb0: Status 404 returned error can't find the container with id 4a2e5b5679eb4518c591962eb63eefd5862e3b6fe68537ba85deba2e89e21bb0 Oct 31 03:41:00.825872 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:00.825839 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" event={"ID":"f7f96aea-6782-4e48-ac8a-019f67f2352c","Type":"ContainerStarted","Data":"4a2e5b5679eb4518c591962eb63eefd5862e3b6fe68537ba85deba2e89e21bb0"} Oct 31 03:41:01.833159 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:01.833115 4456 generic.go:356] "Generic (PLEG): container finished" podID="f7f96aea-6782-4e48-ac8a-019f67f2352c" containerID="8b0842ed551959b11a8b9bd424d8a975c31466e88ac4bbf7612637df580c19c8" exitCode=0 Oct 31 03:41:01.833577 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:01.833206 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" event={"ID":"f7f96aea-6782-4e48-ac8a-019f67f2352c","Type":"ContainerDied","Data":"8b0842ed551959b11a8b9bd424d8a975c31466e88ac4bbf7612637df580c19c8"} Oct 31 03:41:02.133001 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:41:02.132964 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:41:02.974343 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:02.974316 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:03.169423 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.169388 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-xks-ca\") pod \"f7f96aea-6782-4e48-ac8a-019f67f2352c\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " Oct 31 03:41:03.169591 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.169476 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f96aea-6782-4e48-ac8a-019f67f2352c-kube-api-access-nghts\" (UniqueName: \"kubernetes.io/projected/f7f96aea-6782-4e48-ac8a-019f67f2352c-kube-api-access-nghts\") pod \"f7f96aea-6782-4e48-ac8a-019f67f2352c\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " Oct 31 03:41:03.169591 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.169514 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-openshift-ca\") pod \"f7f96aea-6782-4e48-ac8a-019f67f2352c\" (UID: \"f7f96aea-6782-4e48-ac8a-019f67f2352c\") " Oct 31 03:41:03.169933 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.169906 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-openshift-ca" pod "f7f96aea-6782-4e48-ac8a-019f67f2352c" (UID: "f7f96aea-6782-4e48-ac8a-019f67f2352c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:03.171371 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.171342 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-xks-ca" pod "f7f96aea-6782-4e48-ac8a-019f67f2352c" (UID: "f7f96aea-6782-4e48-ac8a-019f67f2352c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:03.171579 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.171551 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f96aea-6782-4e48-ac8a-019f67f2352c-kube-api-access-nghts" pod "f7f96aea-6782-4e48-ac8a-019f67f2352c" (UID: "f7f96aea-6782-4e48-ac8a-019f67f2352c"). InnerVolumeSpecName "kube-api-access-nghts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:03.271049 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.271007 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:03.271232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.271072 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nghts\" (UniqueName: \"kubernetes.io/projected/f7f96aea-6782-4e48-ac8a-019f67f2352c-kube-api-access-nghts\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:03.271232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.271083 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f96aea-6782-4e48-ac8a-019f67f2352c-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:03.846241 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.846203 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" event={"ID":"f7f96aea-6782-4e48-ac8a-019f67f2352c","Type":"ContainerDied","Data":"4a2e5b5679eb4518c591962eb63eefd5862e3b6fe68537ba85deba2e89e21bb0"} Oct 31 03:41:03.846241 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.846237 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rlq5j" Oct 31 03:41:03.846469 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:03.846253 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a2e5b5679eb4518c591962eb63eefd5862e3b6fe68537ba85deba2e89e21bb0" Oct 31 03:41:12.027670 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:12.027635 4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml"] Oct 31 03:41:12.030501 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:12.030473 4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efe1033-8grml"] Oct 31 03:41:12.435032 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:12.434987 4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d021a2f4-b742-4017-9b00-3f9ca6108bd8" path="/var/lib/kubelet/pods/d021a2f4-b742-4017-9b00-3f9ca6108bd8/volumes" Oct 31 03:41:16.114001 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.113963 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq"] Oct 31 03:41:16.114418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.114408 4456 cpu_manager.go:414] "containerMap: removing container" podUID="f7f96aea-6782-4e48-ac8a-019f67f2352c" containerName="revoke-keys" Oct 31 03:41:16.114461 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.114419 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f96aea-6782-4e48-ac8a-019f67f2352c" containerName="revoke-keys" Oct 31 03:41:16.114500 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.114494 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f96aea-6782-4e48-ac8a-019f67f2352c" containerName="revoke-keys" Oct 31 03:41:16.118098 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.118080 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.124153 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.124132 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:41:16.126177 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.126154 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kljfr\"" Oct 31 03:41:16.129030 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.129006 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq"] Oct 31 03:41:16.218730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.218698 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqz52\" (UniqueName: \"kubernetes.io/projected/820de695-1734-4d8d-a59f-519d9fcbebc5-kube-api-access-vqz52\") pod \"maas-api-revoke-sub-0165d7d2-vcfmq\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.218730 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.218732 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcfmq\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.218951 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.218769 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcfmq\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.319418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.319382 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqz52\" (UniqueName: \"kubernetes.io/projected/820de695-1734-4d8d-a59f-519d9fcbebc5-kube-api-access-vqz52\") pod \"maas-api-revoke-sub-0165d7d2-vcfmq\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.319418 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.319422 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcfmq\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.319631 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.319470 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcfmq\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.320305 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.320274 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcfmq\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.321864 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.321843 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcfmq\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.332870 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.332848 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqz52\" (UniqueName: \"kubernetes.io/projected/820de695-1734-4d8d-a59f-519d9fcbebc5-kube-api-access-vqz52\") pod \"maas-api-revoke-sub-0165d7d2-vcfmq\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.433024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.432997 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:16.706953 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.706926 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq"] Oct 31 03:41:16.708084 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:41:16.708058 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820de695_1734_4d8d_a59f_519d9fcbebc5.slice/crio-a31a95d58a264dd918a28539dd1b25f0372694d8a7ac5747e073f39247c5cb21 WatchSource:0}: Error finding container a31a95d58a264dd918a28539dd1b25f0372694d8a7ac5747e073f39247c5cb21: Status 404 returned error can't find the container with id a31a95d58a264dd918a28539dd1b25f0372694d8a7ac5747e073f39247c5cb21 Oct 31 03:41:16.934271 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:16.934237 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" event={"ID":"820de695-1734-4d8d-a59f-519d9fcbebc5","Type":"ContainerStarted","Data":"a31a95d58a264dd918a28539dd1b25f0372694d8a7ac5747e073f39247c5cb21"} Oct 31 03:41:17.942456 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:17.942421 4456 generic.go:356] "Generic (PLEG): container finished" podID="820de695-1734-4d8d-a59f-519d9fcbebc5" containerID="27213f522bc08b299bd81c9e3f3b02d59f531c873e80855d50cb689d3be84f63" exitCode=0 Oct 31 03:41:17.942865 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:17.942487 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" event={"ID":"820de695-1734-4d8d-a59f-519d9fcbebc5","Type":"ContainerDied","Data":"27213f522bc08b299bd81c9e3f3b02d59f531c873e80855d50cb689d3be84f63"} Oct 31 03:41:19.079946 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.079912 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:19.138919 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.138886 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-openshift-ca\") pod \"820de695-1734-4d8d-a59f-519d9fcbebc5\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " Oct 31 03:41:19.139122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.138942 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820de695-1734-4d8d-a59f-519d9fcbebc5-kube-api-access-vqz52\" (UniqueName: \"kubernetes.io/projected/820de695-1734-4d8d-a59f-519d9fcbebc5-kube-api-access-vqz52\") pod \"820de695-1734-4d8d-a59f-519d9fcbebc5\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " Oct 31 03:41:19.139122 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.139014 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-xks-ca\") pod \"820de695-1734-4d8d-a59f-519d9fcbebc5\" (UID: \"820de695-1734-4d8d-a59f-519d9fcbebc5\") " Oct 31 03:41:19.139341 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.139316 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-openshift-ca" pod "820de695-1734-4d8d-a59f-519d9fcbebc5" (UID: "820de695-1734-4d8d-a59f-519d9fcbebc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:19.141181 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.141157 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-xks-ca" pod "820de695-1734-4d8d-a59f-519d9fcbebc5" (UID: "820de695-1734-4d8d-a59f-519d9fcbebc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:19.141278 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.141257 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820de695-1734-4d8d-a59f-519d9fcbebc5-kube-api-access-vqz52" pod "820de695-1734-4d8d-a59f-519d9fcbebc5" (UID: "820de695-1734-4d8d-a59f-519d9fcbebc5"). InnerVolumeSpecName "kube-api-access-vqz52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:19.239584 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.239512 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqz52\" (UniqueName: \"kubernetes.io/projected/820de695-1734-4d8d-a59f-519d9fcbebc5-kube-api-access-vqz52\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:19.239584 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.239539 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:19.239584 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.239548 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/820de695-1734-4d8d-a59f-519d9fcbebc5-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:19.954956 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.954920 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" Oct 31 03:41:19.955162 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.954960 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcfmq" event={"ID":"820de695-1734-4d8d-a59f-519d9fcbebc5","Type":"ContainerDied","Data":"a31a95d58a264dd918a28539dd1b25f0372694d8a7ac5747e073f39247c5cb21"} Oct 31 03:41:19.955162 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:19.955048 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31a95d58a264dd918a28539dd1b25f0372694d8a7ac5747e073f39247c5cb21" Oct 31 03:41:26.980917 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:26.980874 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8"] Oct 31 03:41:26.981560 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:26.981537 4456 cpu_manager.go:414] "containerMap: removing container" podUID="820de695-1734-4d8d-a59f-519d9fcbebc5" containerName="revoke-keys" Oct 31 03:41:26.981664 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:26.981562 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820de695-1734-4d8d-a59f-519d9fcbebc5" containerName="revoke-keys" Oct 31 03:41:26.981722 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:26.981662 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="820de695-1734-4d8d-a59f-519d9fcbebc5" containerName="revoke-keys" Oct 31 03:41:26.985271 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:26.985247 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:26.990422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:26.990390 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:41:26.996734 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:26.996313 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kljfr\"" Oct 31 03:41:26.998237 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:26.998209 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8"] Oct 31 03:41:27.006682 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.006651 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v8lp8\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:27.006829 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.006804 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v8lp8\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:27.006922 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.006905 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s2vj\" (UniqueName: \"kubernetes.io/projected/55c340e1-de1f-468b-bfcc-9dfa53380f8f-kube-api-access-4s2vj\") pod \"maas-api-revoke-sub-6b19338f-v8lp8\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:27.107851 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.107812 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2vj\" (UniqueName: \"kubernetes.io/projected/55c340e1-de1f-468b-bfcc-9dfa53380f8f-kube-api-access-4s2vj\") pod \"maas-api-revoke-sub-6b19338f-v8lp8\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:27.108110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.107865 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v8lp8\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:27.108110 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.107923 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v8lp8\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:27.108698 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.108670 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v8lp8\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:27.110410 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.110384 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v8lp8\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:27.121435 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.121405 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s2vj\" (UniqueName: \"kubernetes.io/projected/55c340e1-de1f-468b-bfcc-9dfa53380f8f-kube-api-access-4s2vj\") pod \"maas-api-revoke-sub-6b19338f-v8lp8\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:27.306322 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.306235 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:27.606499 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:27.606468 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8"] Oct 31 03:41:27.607484 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:41:27.607455 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c340e1_de1f_468b_bfcc_9dfa53380f8f.slice/crio-29ca7b55158482bf20ca4ccb2c82e8f866b8bf6ba77a0c2b66b1f937dc21c633 WatchSource:0}: Error finding container 29ca7b55158482bf20ca4ccb2c82e8f866b8bf6ba77a0c2b66b1f937dc21c633: Status 404 returned error can't find the container with id 29ca7b55158482bf20ca4ccb2c82e8f866b8bf6ba77a0c2b66b1f937dc21c633 Oct 31 03:41:28.008643 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:28.008602 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" event={"ID":"55c340e1-de1f-468b-bfcc-9dfa53380f8f","Type":"ContainerStarted","Data":"364cf5ca52d8f9fc088bde80e98b72700b5160cf3187db31f8927e3d3b1c45f2"} Oct 31 03:41:28.009058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:28.008667 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" event={"ID":"55c340e1-de1f-468b-bfcc-9dfa53380f8f","Type":"ContainerStarted","Data":"29ca7b55158482bf20ca4ccb2c82e8f866b8bf6ba77a0c2b66b1f937dc21c633"} Oct 31 03:41:28.031493 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:28.031447 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" podStartSLOduration=2.031432961 podStartE2EDuration="2.031432961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:28.030545287 +0000 UTC m=+2769.711071414" watchObservedRunningTime="2026-10-31 03:41:28.031432961 +0000 UTC m=+2769.711959088" Oct 31 03:41:29.016244 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:29.016207 4456 generic.go:356] "Generic (PLEG): container finished" podID="55c340e1-de1f-468b-bfcc-9dfa53380f8f" containerID="364cf5ca52d8f9fc088bde80e98b72700b5160cf3187db31f8927e3d3b1c45f2" exitCode=0 Oct 31 03:41:29.016703 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:29.016297 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" event={"ID":"55c340e1-de1f-468b-bfcc-9dfa53380f8f","Type":"ContainerDied","Data":"364cf5ca52d8f9fc088bde80e98b72700b5160cf3187db31f8927e3d3b1c45f2"} Oct 31 03:41:30.148105 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:30.148080 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:30.235567 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:30.235534 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-xks-ca\") pod \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " Oct 31 03:41:30.235736 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:30.235640 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c340e1-de1f-468b-bfcc-9dfa53380f8f-kube-api-access-4s2vj\" (UniqueName: \"kubernetes.io/projected/55c340e1-de1f-468b-bfcc-9dfa53380f8f-kube-api-access-4s2vj\") pod \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " Oct 31 03:41:30.235736 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:30.235677 4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-openshift-ca\") pod \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\" (UID: \"55c340e1-de1f-468b-bfcc-9dfa53380f8f\") " Oct 31 03:41:30.236095 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:30.236067 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-openshift-ca" pod "55c340e1-de1f-468b-bfcc-9dfa53380f8f" (UID: "55c340e1-de1f-468b-bfcc-9dfa53380f8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:30.237633 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:30.237603 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-xks-ca" pod "55c340e1-de1f-468b-bfcc-9dfa53380f8f" (UID: "55c340e1-de1f-468b-bfcc-9dfa53380f8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:30.237743 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:30.237646 4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c340e1-de1f-468b-bfcc-9dfa53380f8f-kube-api-access-4s2vj" pod "55c340e1-de1f-468b-bfcc-9dfa53380f8f" (UID: "55c340e1-de1f-468b-bfcc-9dfa53380f8f"). InnerVolumeSpecName "kube-api-access-4s2vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:30.336604 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:30.336520 4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s2vj\" (UniqueName: \"kubernetes.io/projected/55c340e1-de1f-468b-bfcc-9dfa53380f8f-kube-api-access-4s2vj\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:30.336604 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:30.336551 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-openshift-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:30.336604 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:30.336562 4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c340e1-de1f-468b-bfcc-9dfa53380f8f-maas-api-xks-ca\") on node \"ip-10-0-129-107.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:31.028558 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:31.028531 4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" Oct 31 03:41:31.028738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:31.028525 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v8lp8" event={"ID":"55c340e1-de1f-468b-bfcc-9dfa53380f8f","Type":"ContainerDied","Data":"29ca7b55158482bf20ca4ccb2c82e8f866b8bf6ba77a0c2b66b1f937dc21c633"} Oct 31 03:41:31.028738 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:31.028679 4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ca7b55158482bf20ca4ccb2c82e8f866b8bf6ba77a0c2b66b1f937dc21c633" Oct 31 03:41:32.132940 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:41:32.132898 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:41:49.489396 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:49.489361 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qprkr_462a67f6-9d05-41d3-81a4-6729996976a1/manager/0.log" Oct 31 03:41:49.881189 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:49.881113 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wwfhp_af4ef643-fdf2-41ab-ae84-82c812b965e6/manager/0.log" Oct 31 03:41:50.774267 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:50.774237 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-72fx4_8e8f2d86-6510-4386-befc-ec0c5c031127/proxy/0.log" Oct 31 03:41:51.997238 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:51.997189 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-vcfmq_820de695-1734-4d8d-a59f-519d9fcbebc5/revoke-keys/0.log" Oct 31 03:41:52.745906 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:52.745872 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-v8lp8_55c340e1-de1f-468b-bfcc-9dfa53380f8f/revoke-keys/0.log" Oct 31 03:41:52.943422 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:52.943392 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-rlq5j_f7f96aea-6782-4e48-ac8a-019f67f2352c/revoke-keys/0.log" Oct 31 03:41:53.129293 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:53.129212 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-9t8b9_7f66efa3-2b56-47c3-9c5a-05b9e597c338/postgres/0.log" Oct 31 03:41:55.145684 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:55.145658 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lblcf_feb0e371-5edb-41d6-a003-9e4adf0853b9/manager/0.log" Oct 31 03:41:55.544893 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:55.544859 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-gjpzm_4e1ff94e-bee2-4e63-beda-55fed25dbfbe/manager/0.log" Oct 31 03:41:57.362525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:57.362483 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d9fdc54f-j5svm_7bfd4f57-1369-425c-a3f4-d762e147c354/kube-auth-proxy/0.log" Oct 31 03:41:58.786395 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:58.786357 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-n6sbp_29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3/istio-proxy/0.log" Oct 31 03:41:59.179625 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:59.179590 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7gdw_4f1f2ef9-d74d-4c50-8eeb-60103414adaa/istio-proxy/0.log" Oct 31 03:41:59.387316 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:41:59.387282 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7r8z_ef33ec8e-50ff-4f0b-a84c-25a9b07c2221/istio-proxy/1.log" Oct 31 03:42:00.608649 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:00.608609 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7r8z_ef33ec8e-50ff-4f0b-a84c-25a9b07c2221/istio-proxy/0.log" Oct 31 03:42:02.135232 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:42:02.135195 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:42:13.360208 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:13.360177 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qprkr_462a67f6-9d05-41d3-81a4-6729996976a1/manager/0.log" Oct 31 03:42:13.814359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:13.814329 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wwfhp_af4ef643-fdf2-41ab-ae84-82c812b965e6/manager/0.log" Oct 31 03:42:14.697969 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:14.697943 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-72fx4_8e8f2d86-6510-4386-befc-ec0c5c031127/proxy/0.log" Oct 31 03:42:15.699279 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:15.699249 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-vcfmq_820de695-1734-4d8d-a59f-519d9fcbebc5/revoke-keys/0.log" Oct 31 03:42:16.546468 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:16.546442 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-v8lp8_55c340e1-de1f-468b-bfcc-9dfa53380f8f/revoke-keys/0.log" Oct 31 03:42:16.744046 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:16.744015 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-rlq5j_f7f96aea-6782-4e48-ac8a-019f67f2352c/revoke-keys/0.log" Oct 31 03:42:16.990401 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:16.990374 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-9t8b9_7f66efa3-2b56-47c3-9c5a-05b9e597c338/postgres/0.log" Oct 31 03:42:18.617917 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:18.617883 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lblcf_feb0e371-5edb-41d6-a003-9e4adf0853b9/manager/0.log" Oct 31 03:42:19.004649 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:19.004613 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-gjpzm_4e1ff94e-bee2-4e63-beda-55fed25dbfbe/manager/0.log" Oct 31 03:42:20.615840 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:20.615809 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d9fdc54f-j5svm_7bfd4f57-1369-425c-a3f4-d762e147c354/kube-auth-proxy/0.log" Oct 31 03:42:20.777755 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:20.777724 4456 scope.go:122] "RemoveContainer" containerID="d437f6e96c2326bd6c54ca486abb98eceb26c55c7900328ae891b22fdb2bf392" Oct 31 03:42:20.787050 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:20.787016 4456 scope.go:122] "RemoveContainer" containerID="6a001ad9e1c3867d573e89dc2fb61f25f460ea41820f85f83f48e3dd5067ea80" Oct 31 03:42:22.157024 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:22.156989 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-n6sbp_29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3/istio-proxy/0.log" Oct 31 03:42:22.578495 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:22.578398 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7gdw_4f1f2ef9-d74d-4c50-8eeb-60103414adaa/istio-proxy/0.log" Oct 31 03:42:22.812659 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:42:22.812631 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7r8z_ef33ec8e-50ff-4f0b-a84c-25a9b07c2221/istio-proxy/1.log" Oct 31 03:42:32.133437 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:42:32.133357 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:43:02.133268 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:43:02.133228 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:43:32.133106 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:43:32.133065 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:43:55.806398 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:55.806356 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-68h4s/must-gather-w9glt"] Oct 31 03:43:55.806811 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:55.806787 4456 cpu_manager.go:414] "containerMap: removing container" podUID="55c340e1-de1f-468b-bfcc-9dfa53380f8f" containerName="revoke-keys" Oct 31 03:43:55.806811 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:55.806799 4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c340e1-de1f-468b-bfcc-9dfa53380f8f" containerName="revoke-keys" Oct 31 03:43:55.806896 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:55.806885 4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c340e1-de1f-468b-bfcc-9dfa53380f8f" containerName="revoke-keys" Oct 31 03:43:55.810265 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:55.810248 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68h4s/must-gather-w9glt" Oct 31 03:43:55.817678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:55.817647 4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-68h4s\"/\"default-dockercfg-sdxsf\"" Oct 31 03:43:55.817678 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:55.817654 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68h4s\"/\"openshift-service-ca.crt\"" Oct 31 03:43:55.817818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:55.817681 4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68h4s\"/\"kube-root-ca.crt\"" Oct 31 03:43:55.830960 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:55.830936 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68h4s/must-gather-w9glt"] Oct 31 03:43:56.000979 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:56.000948 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7153af1f-e389-4757-92d8-0760aee8391b-must-gather-output\") pod \"must-gather-w9glt\" (UID: \"7153af1f-e389-4757-92d8-0760aee8391b\") " pod="openshift-must-gather-68h4s/must-gather-w9glt" Oct 31 03:43:56.001172 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:56.001012 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwt5s\" (UniqueName: \"kubernetes.io/projected/7153af1f-e389-4757-92d8-0760aee8391b-kube-api-access-wwt5s\") pod \"must-gather-w9glt\" (UID: \"7153af1f-e389-4757-92d8-0760aee8391b\") " pod="openshift-must-gather-68h4s/must-gather-w9glt" Oct 31 03:43:56.102518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:56.102409 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7153af1f-e389-4757-92d8-0760aee8391b-must-gather-output\") pod \"must-gather-w9glt\" (UID: \"7153af1f-e389-4757-92d8-0760aee8391b\") " pod="openshift-must-gather-68h4s/must-gather-w9glt" Oct 31 03:43:56.102518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:56.102498 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwt5s\" (UniqueName: \"kubernetes.io/projected/7153af1f-e389-4757-92d8-0760aee8391b-kube-api-access-wwt5s\") pod \"must-gather-w9glt\" (UID: \"7153af1f-e389-4757-92d8-0760aee8391b\") " pod="openshift-must-gather-68h4s/must-gather-w9glt" Oct 31 03:43:56.102863 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:56.102838 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7153af1f-e389-4757-92d8-0760aee8391b-must-gather-output\") pod \"must-gather-w9glt\" (UID: \"7153af1f-e389-4757-92d8-0760aee8391b\") " pod="openshift-must-gather-68h4s/must-gather-w9glt" Oct 31 03:43:56.116031 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:56.116000 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwt5s\" (UniqueName: \"kubernetes.io/projected/7153af1f-e389-4757-92d8-0760aee8391b-kube-api-access-wwt5s\") pod \"must-gather-w9glt\" (UID: \"7153af1f-e389-4757-92d8-0760aee8391b\") " pod="openshift-must-gather-68h4s/must-gather-w9glt" Oct 31 03:43:56.124892 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:56.124871 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68h4s/must-gather-w9glt" Oct 31 03:43:56.402432 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:43:56.402402 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7153af1f_e389_4757_92d8_0760aee8391b.slice/crio-0516a808076aceab32b9db345a1ecd17ac6734cd101c43e6f7dcc2837edb4f35 WatchSource:0}: Error finding container 0516a808076aceab32b9db345a1ecd17ac6734cd101c43e6f7dcc2837edb4f35: Status 404 returned error can't find the container with id 0516a808076aceab32b9db345a1ecd17ac6734cd101c43e6f7dcc2837edb4f35 Oct 31 03:43:56.402578 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:56.402552 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68h4s/must-gather-w9glt"] Oct 31 03:43:56.404296 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:56.404276 4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:43:56.870818 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:56.870787 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68h4s/must-gather-w9glt" event={"ID":"7153af1f-e389-4757-92d8-0760aee8391b","Type":"ContainerStarted","Data":"0516a808076aceab32b9db345a1ecd17ac6734cd101c43e6f7dcc2837edb4f35"} Oct 31 03:43:57.883519 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:57.883481 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68h4s/must-gather-w9glt" event={"ID":"7153af1f-e389-4757-92d8-0760aee8391b","Type":"ContainerStarted","Data":"3553ccd5bb7f10d75a2d9f13925032580591452fead85b027fb63eedb2f02a7f"} Oct 31 03:43:57.884067 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:57.883531 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68h4s/must-gather-w9glt" event={"ID":"7153af1f-e389-4757-92d8-0760aee8391b","Type":"ContainerStarted","Data":"60579574c3a99ff19c89e5d08490b6062282e55b7f32717a695ba8f6dddc94f7"} Oct 31 03:43:57.909222 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:57.909164 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-68h4s/must-gather-w9glt" podStartSLOduration=1.9949324819999998 podStartE2EDuration="2.909146256s" totalImagesPullingTime="914.213774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:57.907931019 +0000 UTC m=+2919.588457147" watchObservedRunningTime="2026-10-31 03:43:57.909146256 +0000 UTC m=+2919.589672384" Oct 31 03:43:59.238604 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:59.238567 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vtlr6_3b7047af-e1c4-4e1c-81ea-f777c15b0d97/global-pull-secret-syncer/0.log" Oct 31 03:43:59.470178 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:59.470143 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-khwpz_24d5a9bc-29a3-42b5-bf90-9468b578b0f7/konnectivity-agent/0.log" Oct 31 03:43:59.556971 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:43:59.556891 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-107.us-east-2.compute.internal_da9a871c29174a296dbd1f4a73e3997d/haproxy/0.log" Oct 31 03:44:02.218593 ip-10-0-129-107 kubenswrapper[4456]: E1031 03:44:02.218519 4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rb8r5" containerName="extractor" Oct 31 03:44:06.003229 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:06.003167 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lblcf_feb0e371-5edb-41d6-a003-9e4adf0853b9/manager/0.log" Oct 31 03:44:06.108525 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:06.108391 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-gjpzm_4e1ff94e-bee2-4e63-beda-55fed25dbfbe/manager/0.log" Oct 31 03:44:07.665076 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:07.662327 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94846398-db50-47bb-8382-30438dc3bf34/alertmanager/0.log" Oct 31 03:44:07.692170 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:07.692145 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94846398-db50-47bb-8382-30438dc3bf34/config-reloader/0.log" Oct 31 03:44:07.722003 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:07.721972 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94846398-db50-47bb-8382-30438dc3bf34/kube-rbac-proxy-web/0.log" Oct 31 03:44:07.754060 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:07.753893 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94846398-db50-47bb-8382-30438dc3bf34/kube-rbac-proxy/0.log" Oct 31 03:44:07.783063 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:07.782923 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94846398-db50-47bb-8382-30438dc3bf34/kube-rbac-proxy-metric/0.log" Oct 31 03:44:07.815518 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:07.815397 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94846398-db50-47bb-8382-30438dc3bf34/prom-label-proxy/0.log" Oct 31 03:44:07.862090 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:07.858808 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94846398-db50-47bb-8382-30438dc3bf34/init-config-reloader/0.log" Oct 31 03:44:07.975974 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:07.975874 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hrd9c_9e6cae46-c89a-458b-93a5-e313e7c01403/kube-state-metrics/0.log" Oct 31 03:44:08.000938 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.000906 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hrd9c_9e6cae46-c89a-458b-93a5-e313e7c01403/kube-rbac-proxy-main/0.log" Oct 31 03:44:08.042058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.041026 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hrd9c_9e6cae46-c89a-458b-93a5-e313e7c01403/kube-rbac-proxy-self/0.log" Oct 31 03:44:08.115059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.114338 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-fzv6z_0e3a2f93-5cb8-4c2b-8096-2c26b0bc4d06/monitoring-plugin/0.log" Oct 31 03:44:08.160059 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.159666 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75lnw_b9c04022-9e9e-4704-986e-ad2ef2993224/node-exporter/0.log" Oct 31 03:44:08.198144 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.196831 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75lnw_b9c04022-9e9e-4704-986e-ad2ef2993224/kube-rbac-proxy/0.log" Oct 31 03:44:08.226642 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.226576 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75lnw_b9c04022-9e9e-4704-986e-ad2ef2993224/init-textfile/0.log" Oct 31 03:44:08.474093 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.474062 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hzk4k_12194edf-58dc-4167-a1e0-8888ccf43cc9/kube-rbac-proxy-main/0.log" Oct 31 03:44:08.506488 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.506401 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hzk4k_12194edf-58dc-4167-a1e0-8888ccf43cc9/kube-rbac-proxy-self/0.log" Oct 31 03:44:08.536748 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.536721 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hzk4k_12194edf-58dc-4167-a1e0-8888ccf43cc9/openshift-state-metrics/0.log" Oct 31 03:44:08.812752 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.812663 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ft948_b54ae00b-53b0-426d-b12a-88a8c03dfc13/prometheus-operator/0.log" Oct 31 03:44:08.841394 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.841364 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ft948_b54ae00b-53b0-426d-b12a-88a8c03dfc13/kube-rbac-proxy/0.log" Oct 31 03:44:08.875634 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.875605 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lmdmj_b7cbf4db-e402-4071-bdaa-7c4ffc6d3626/prometheus-operator-admission-webhook/0.log" Oct 31 03:44:08.912571 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.912535 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f56555f87-62h7x_2177bb5e-812d-4873-830a-2ab7e339c610/telemeter-client/0.log" Oct 31 03:44:08.944994 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.944966 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f56555f87-62h7x_2177bb5e-812d-4873-830a-2ab7e339c610/reload/0.log" Oct 31 03:44:08.975087 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:08.975028 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f56555f87-62h7x_2177bb5e-812d-4873-830a-2ab7e339c610/kube-rbac-proxy/0.log" Oct 31 03:44:09.033723 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.033593 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-678874f684-2zvzr_ebaf5bff-0da3-496e-8461-914f5f374daf/thanos-query/0.log" Oct 31 03:44:09.059896 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.059850 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-678874f684-2zvzr_ebaf5bff-0da3-496e-8461-914f5f374daf/kube-rbac-proxy-web/0.log" Oct 31 03:44:09.092215 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.092134 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-678874f684-2zvzr_ebaf5bff-0da3-496e-8461-914f5f374daf/kube-rbac-proxy/0.log" Oct 31 03:44:09.120859 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.120831 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-678874f684-2zvzr_ebaf5bff-0da3-496e-8461-914f5f374daf/prom-label-proxy/0.log" Oct 31 03:44:09.154681 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.154652 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-678874f684-2zvzr_ebaf5bff-0da3-496e-8461-914f5f374daf/kube-rbac-proxy-rules/0.log" Oct 31 03:44:09.172997 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.172963 4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7"] Oct 31 03:44:09.181150 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.181120 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.189958 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.189934 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-678874f684-2zvzr_ebaf5bff-0da3-496e-8461-914f5f374daf/kube-rbac-proxy-metrics/0.log" Oct 31 03:44:09.191572 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.191545 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7"] Oct 31 03:44:09.272064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.270535 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-lib-modules\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.272064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.270627 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-podres\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.272064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.270667 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-proc\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.272064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.270716 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rxsc\" (UniqueName: \"kubernetes.io/projected/7fbb65e2-de9c-405a-a5bd-c587cdab3118-kube-api-access-9rxsc\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.272064 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.270772 4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-sys\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.372101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.371771 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-lib-modules\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.372101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.371851 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-podres\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.372101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.371882 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-proc\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.372101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.371908 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-lib-modules\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.372101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.371935 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rxsc\" (UniqueName: \"kubernetes.io/projected/7fbb65e2-de9c-405a-a5bd-c587cdab3118-kube-api-access-9rxsc\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.372101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.372028 4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-sys\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.372101 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.372098 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-sys\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.372453 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.372027 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-proc\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.372453 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.372155 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7fbb65e2-de9c-405a-a5bd-c587cdab3118-podres\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.385782 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.385753 4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rxsc\" (UniqueName: \"kubernetes.io/projected/7fbb65e2-de9c-405a-a5bd-c587cdab3118-kube-api-access-9rxsc\") pod \"perf-node-gather-daemonset-cvfr7\" (UID: \"7fbb65e2-de9c-405a-a5bd-c587cdab3118\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.499966 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.499931 4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:09.799079 ip-10-0-129-107 kubenswrapper[4456]: W1031 03:44:09.794543 4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7fbb65e2_de9c_405a_a5bd_c587cdab3118.slice/crio-518463cbf45518b2bcfb7b2aea158b317311db80851ebd48bb34fb29fc7d6de4 WatchSource:0}: Error finding container 518463cbf45518b2bcfb7b2aea158b317311db80851ebd48bb34fb29fc7d6de4: Status 404 returned error can't find the container with id 518463cbf45518b2bcfb7b2aea158b317311db80851ebd48bb34fb29fc7d6de4 Oct 31 03:44:09.799079 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:09.796123 4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7"] Oct 31 03:44:10.065234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:10.065130 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" event={"ID":"7fbb65e2-de9c-405a-a5bd-c587cdab3118","Type":"ContainerStarted","Data":"a2bcf3b050c52f3ad97fb08d28d7910b701f8b00670759e4a69feade06374c25"} Oct 31 03:44:10.065234 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:10.065204 4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" event={"ID":"7fbb65e2-de9c-405a-a5bd-c587cdab3118","Type":"ContainerStarted","Data":"518463cbf45518b2bcfb7b2aea158b317311db80851ebd48bb34fb29fc7d6de4"} Oct 31 03:44:10.065747 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:10.065363 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:10.090794 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:10.090732 4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" podStartSLOduration=1.090713047 podStartE2EDuration="1.090713047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:10.08896881 +0000 UTC m=+2931.769494938" watchObservedRunningTime="2026-10-31 03:44:10.090713047 +0000 UTC m=+2931.771239175" Oct 31 03:44:11.727211 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:11.727167 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6769478768-6p8zj_6516fb64-f047-458c-83d8-8a3c824aed5d/console/0.log" Oct 31 03:44:11.780510 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:11.780478 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-kqjnj_31079682-23c1-4203-b654-3a9c93b56901/download-server/0.log" Oct 31 03:44:13.657361 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:13.657329 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hwt9g_810e723f-fc3a-4477-a30d-dc086bd338b1/dns/0.log" Oct 31 03:44:13.687058 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:13.686995 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hwt9g_810e723f-fc3a-4477-a30d-dc086bd338b1/kube-rbac-proxy/0.log" Oct 31 03:44:13.895245 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:13.895220 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mxcvd_ee641d86-9b04-4f02-a22b-7cc55694a018/dns-node-resolver/0.log" Oct 31 03:44:14.590104 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:14.590071 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xlsrl_2be3a160-6ada-4e51-a6bf-19745a8670f4/node-ca/0.log" Oct 31 03:44:16.087205 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:16.087170 4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-cvfr7" Oct 31 03:44:19.964636 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:19.964605 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d9fdc54f-j5svm_7bfd4f57-1369-425c-a3f4-d762e147c354/kube-auth-proxy/0.log" Oct 31 03:44:20.210099 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:20.210067 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-n6sbp_29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3/istio-proxy/0.log" Oct 31 03:44:20.283797 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:20.283725 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7gdw_4f1f2ef9-d74d-4c50-8eeb-60103414adaa/istio-proxy/0.log" Oct 31 03:44:20.312900 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:20.312859 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7r8z_ef33ec8e-50ff-4f0b-a84c-25a9b07c2221/istio-proxy/0.log" Oct 31 03:44:20.313655 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:20.313635 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7r8z_ef33ec8e-50ff-4f0b-a84c-25a9b07c2221/istio-proxy/1.log" Oct 31 03:44:22.213232 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:22.213188 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cqgsq_b4bb32ff-188e-4471-8c62-08fae7e26c8c/serve-healthcheck-canary/0.log" Oct 31 03:44:22.878000 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:22.877965 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rb8r5_4ded5081-b545-47f2-be60-975cf616f05f/kube-rbac-proxy/0.log" Oct 31 03:44:22.906359 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:22.906331 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rb8r5_4ded5081-b545-47f2-be60-975cf616f05f/exporter/0.log" Oct 31 03:44:22.935274 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:22.935246 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rb8r5_4ded5081-b545-47f2-be60-975cf616f05f/extractor/0.log" Oct 31 03:44:23.493196 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:23.493165 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d9fdc54f-j5svm_7bfd4f57-1369-425c-a3f4-d762e147c354/kube-auth-proxy/0.log" Oct 31 03:44:23.646609 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:23.646562 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-n6sbp_29bd1c55-1ba5-4ab2-9d5a-e4d41d3daec3/istio-proxy/0.log" Oct 31 03:44:23.691813 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:23.691781 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7gdw_4f1f2ef9-d74d-4c50-8eeb-60103414adaa/istio-proxy/0.log" Oct 31 03:44:23.709544 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:23.709513 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7r8z_ef33ec8e-50ff-4f0b-a84c-25a9b07c2221/istio-proxy/0.log" Oct 31 03:44:23.713406 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:23.713375 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7r8z_ef33ec8e-50ff-4f0b-a84c-25a9b07c2221/istio-proxy/1.log" Oct 31 03:44:25.571620 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:25.571576 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-57b855755-j7fhl_573ec2a0-2d75-479d-a447-1804dedf242a/manager/0.log" Oct 31 03:44:25.601687 ip-10-0-129-107 kubenswrapper[4456]: I1031 03:44:25.601655 4456 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-57b855755-j7fhl_573ec2a0-2d75-479d-a447-1804dedf242a/copy-manifests/0.log"